From Data to services Building services from the Web : Open data and semantic data sources Muriel Foulonneau, Géraldine Vidou muriel.foulonneau@tudor.lu.

From Data to services Building services from the Web : Open data and semantic data sources Muriel Foulonneau, Géraldine Vidou [email protected].

Transcript From Data to services Building services from the Web : Open data and semantic data sources Muriel Foulonneau, Géraldine Vidou [email protected].

From Data to services

Building services from the Web : Open data and semantic data sources

Muriel Foulonneau, Géraldine Vidou

The Tudor Public Research Centre KISS group, Service Science Innovation department

 Research and innovation on Knowledge technologies  A multi-disciplinary team, including psychologists, experts in semantic technologies, software engineering, modelling, data mining, tangible interfaces  Application domains: eGovernment, educational assessment, recommender systems, logistics  Involved in many international projects, OECD, FP7, ICT PSP, EUREKA, CELTIC ….

 Industrial partnerships, in Luxembourg (BCE), in Europe (Nokia, Thomson), in the US (ETS)  From R&D to spin off 29/04/2020 Presentation Tudor 2

3 parts Part I- Data Part II- Example services based on data Part III- Designing services based on data

29/04/2020 3

29/04/2020

Part I- Data

Presentation Tudor 4

29/04/2020

TO DO What is data?

Presentation Tudor 5

Several definitions “Data are the small snippets of information that people collect, join together in data records, and store in databases.” (Boiko, 2001). Things that can be processed by computers Gadomski (1999) what is necessary to make a selection between two possible options (Brillouin, 1962 ) (based on Shannon’s information theory) “simple observations of states of the world”, “symbols which have not yet been interpreted” (Tuomi, 2000) “Data, therefore, exists as a solution to a practical problem: how to dissect information into two forms, data and data structure, that can be modeled, represented, and processed separately. As the computer does not have access to the meaning of the content it processes, computer programmers have to represent meaning in a way that makes automatic processing possible.” (Tuomi, 2000).

29/04/2020 6

The knowledge pyramid

http://www.systems-thinking.org/dikw/dikw.htm

“data are assumed to be simple isolated facts. When such facts are put into a context, and combined within a structure, information emerges. When information is given meaning by interpreting it, information becomes knowledge.” (Tuomi, 2000) 29/04/2020 Paul Beynon-Davies, Neolithic informatics: The nature of information, International Journal of Information Management 29 (2009) 3–14 7

Data and its context

29/04/2020 8

Data and the knowledge pyramid otherwise illustrated?

29/04/2020 http://e-reality-home.blogspot.com/2008/03/origin-of-dikw hierarchy.html

“Image originally published in the December 1982 issue of THE FUTURIST. Used with permission from the World Future Society, 7910 Woodmont Avenue, Suite 450, Bethesda, Maryland 20814. Telephone: 301/656-8274; Fax: 301/951-0394; http://www.wfs.org

” 9

Services to make sense of data or to use data The multiplication of data (and content) sources Examples of data sources Some keywords: Open Data, Linked Data, semantic data sources, data mining, data processing, data quality Examples of services Some keywords: data competitions (hackdays etc), service innovation cycle, business model From data to services: process and challenges Some keywords: innovation, governance, sustainability

29/04/2020 10

29/04/2020

Open Data

Presentation Tudor 11

Data openness 1. Access 2. Redistribution 3. Reuse 4. Absence of Technological Restriction 5. Attribution 6. Integrity 7. No Discrimination Against Persons or Groups 8. No Discrimination Against Fields of Endeavor 9. Distribution of License 10. License Must Not Be Specific to a Package 11. License Must Not Restrict the Distribution of Other Works

http://opendefinition.org/okd/ 29/04/2020 12

Why opening data?

Political impact transparency and democracy Social impact create many services: smart cities Business impact business opportunity

29/04/2020 13

Some tools

CKAN: Comprehensive Knowledge Archive Network

OKF tool to create open data catalogues

DCAT: Data Catalog Vocabulary

Creative Common licences

http://ckan.org/features/ 14 29/04/2020

Open Data in Europe The Public Data EU catalogue (CKAN)

29/04/2020 15

Licenses

29/04/2020 16

Data format

29/04/2020 17

Actual openness of data Based on their metadata, 75% of the datasets can be stated to be open at any level

29/04/2020 18

Data Publica The study relies on Data Publica rather than data.gouv.fr (353.000+ datasets)



Semi-private company



Data store



Data as a Service



Data Tuesday events



Scrapathon : collect and transform data



http://www.slate.fr/life/73113/scrapathon-data-publica-utiliser-donnees publiques

29/04/2020 Presentation Tudor 19

Data.gov.uk

29/04/2020 20

Berlin Open Data platform

29/04/2020 21

Rennes

29/04/2020 22

Call for data opening interest E.g., the French ministry of culture launched a public consultation on which cultural data should be open => Public demand / potential for service creation

29/04/2020 23

29/04/2020

Semantic Web and Web of data

Presentation Tudor 24

Exemple of a Wikipedia page

29/04/2020 Presentation Tudor 25

Representation on DBpedia

29/04/2020 dbpedia.org/resource/Albert_Einstein Presentation Tudor 26

RDF version of the Dbpedia page

29/04/2020 Presentation Tudor 27

RDF and semantic modeling of data RDF

Einstein

http://me/hasBirthPlace

Ulm

Classes and instances

Me:Location

rdfs:subclass_of

Me:Scientist

rdf:type http://me/hasBirthPlace

Einstein Me:City

rdf:type

Ulm 28

The ability to reason on data Built in inference mechanisms e.g. Ulm is a place Inference rules

Me:Location

rdfs:subclass_of

Me:City

rdf:type

Ulm 29/04/2020 29

Semantic representation of data RDF data

•

Subject – Predicate - Object Models

•

Classes of resources URIs (identifiers) for abstract or digital resources

•

No ambiguity Inference mechanisms

•

So that computers can reason on content

Links across data sets (Linked Data)

Me: Einstein Me: Einstein

http://me/hasBirthPlace

Dbpedia:Ulm

http://me/hasBirthPlace

Me:Ulm

Owl:sameAs

Dbpedia:Ulm 31

Distinct datasets

29/04/2020 32

Graph fusion: data linkage

29/04/2020 33

External references and description of properties

The Linked Data Cloud

Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/ 29/04/2020 ESWC 2011 35

Some statistics on the cloud

29/04/2020 http://lod-cloud.net/state/ 36

Reusing vocabularies A vocabulary registry (LOV) A search engine (Sindice)

29/04/2020 37

Retrieving information on the Web of data SPARQL Keywords

29/04/2020 38

Manual data linkage

29/04/2020 39

Wikidata: making the Web of data social

29/04/2020 40

Semantic data sources also use many semantic formalisms

http://ogp.me/ http://schema.org/docs/gs.html#schemaorg_types http://en.wikipedia.org/wiki/JSON 29/04/2020 41

29/04/2020

TO DO What is reusable data?

Presentation Tudor 42

Part II- Examples of service design based on data

29/04/2020 Presentation Tudor 43

Example 1 Designing a portal using OAI data

e.g., Europeana, CIC metadata portal

29/04/2020 Presentation Tudor 44

Example portals

29/04/2020 45

Sharing metadata: federation approach The user queries the system for “mill”, answer whichever the repository

My resource</t itle> <date>04 mill? <title>My resource</t itle> <date>04 <title>My resource</t itle> <date>04 E.g., Z39.50, SRU/SRW muriel.foulonneau@tudor.lu <a id="p47"></a> Metadata aggregation Asynchronous model <title>My resource</t itle> <date>04 Search engines, Uniion Catalogues, Open Archives Initiative Protocol for Metadata Harvesting muriel.foulonneau@tudor.lu <a id="p48"></a> OAI-PMH : “Divides the world between data providers and service providers” Service provider Portal interface Harvester Repository Repository Repository Data provider Data provider Agregator Data provider Data provider  Mechanism to transfer large quantities of data   Incremental harvests Resumption tokens muriel.foulonneau@tudor.lu <a id="p49"></a> Incremental harvest Data provider What’s new since I last came? Harvester <title>My resource</t itle> <date>04 •New data, updated data •Deleted data muriel.foulonneau@tudor.lu 49 <a id="p50"></a> Multiple representation of resources oai:lcoa1.loc.gov:loc.pnp/ cph.3b23005 Dublin Core MARC21 MODS Displayed record School of arts for girls Kiz Sanayi Mektebi] muriel.foulonneau@tudor.lu <a id="p51"></a> Coexistence de various services Plein texte Métadonnées et ressources Métadonnées Desc. collections muriel.foulonneau@tudor.lu Métadonnées et ressources Métadonnées <a id="p52"></a> Various levels of interoperability Technical Protocoles , material, … Mac/PC, Firefox/IE… Organisational • Responsibilties, competencies, strategies, updates, business rules, processes Content related: metadata • • • • Descriptive unit (granularity and nature) Semantic : date…. Creation date? Publication date? Syntactic : 04 January 2004 Linguistic : 04 Enero 2004 muriel.foulonneau@tudor.lu <a id="p53"></a> <h3>Data aggregation in practice</h3> muriel.foulonneau@tudor.lu 53 <a id="p54"></a> Data reprocessing archiving Data collection Record selection Cleaning E.g.: exclude if no digital object – according to collection development policy / service policy E.g.: delete if null values <dc:format/> <dc:date>--</dc:date> Harmonizing / mapping Enriching E.g.: mapping Format – Extent when value= « 163 pages » E.g.: add DCMIType, add provenance dataset muriel.foulonneau@tudor.lu <a id="p55"></a> Harmonizing / enriching data Concept Type Format Language Collection Resource URL Date : 71% Localisation : 21% muriel.foulonneau@tudor.lu % enregistrements CIC 99% 66% 60% 100% 72% <a id="p56"></a> Data accessibility Who is the data intended to? • machine • • Dc:type “Text.Correspondence.Letter” Dc:language “wln” • human • • Dc:type Correspondence Dc:language “wallon” • ? • • Dc:date “197- “ Dc:description “First ed. Cf. BM. “ DC:Type = Identification photographs DC:Type = StillImage DC:Type = Image muriel.foulonneau@tudor.lu <a id="p57"></a> Loss of data context http://commons.wikimedia.org/wiki/File:President_Roosevelt_on_a_horse.jpg muriel.foulonneau@tudor.lu <a id="p58"></a> What’s behind URLs muriel.foulonneau@tudor.lu <a id="p59"></a> From metadata to digital resources # clicks between service and digital resource 1click 2 clicks 3 clicks Total # collections (considering thumbnail and table of content as digital resources) # collections (without considering thumbnail and table of content as digital resources) 54 26 39 3 96 53 17 96 Out of 96 collections (CIC, 06/2006) muriel.foulonneau@tudor.lu <a id="p60"></a> Hetrogeneous data creation practices http://www.oaforum.org/otherfiles/bath_cole.ppt muriel.foulonneau@tudor.lu 60 <a id="p61"></a> Language issues e.g., a record in 4 languages oai:images.library.uiuc.edu:RussianPublics/8 < oai_dc:dc > < title > Moloshnitsa i Prachka. La laitière et la blanchisseuse. Das milchweib und die Wäscherinn. </ title > < description > (unknown) </ description > < creator > (unknown) </ creator > < subject /> < source > Volshebnoi fonar' </ source > < description 0. </ > Image was captured using Epson ### scanner and SilverFast import utility in Adobe Photoshop 6. Scanned at 42->24 Bit Colour at 300 dpi. No archival images were created, but access images were created as JPEGs with quality of description > < rights > Contact the owner of the material for copyright information. If you have comments relating to this record, please contact the Slavic and East European Library, http://www.library.uiuc.edu/spx/. </ rights > < identifier > http://images.library.uiuc.edu:8081/u?/RussianPublics,8 </ identifier > </ oai_dc:dc > muriel.foulonneau@tudor.lu 61 <a id="p62"></a> Examples of values found as DC:Date September 29 –October 28, 51 AD; 1970 second half of IXth century AD; 1978 Rebuilt 1984 Possibly Vth/VIth century AD; 1935 Planted 1985 n/a n.d. Mid IInd century AD; 1973 Jul-51 circa 900 AD ca. 701 BC Begun 14th century 184-? 1839 18 –? August 23, 2000 between 1827 and 183 VIIIth/IXth century AD ? (TC);1965 Vth-VIth century AD (McNamee); IVth century AD (Cribiore); 1982 XVIII Dynasty Winter 2003 era of redevelopment various 2002-00 1980, refurbished 1997 China: Neolithic Period (5000 BCE-ca 1600 BCE)? 19691968 21. Nouemb. Anno. 1564 And finisshed on the euen of thanunciacion of our said bilissid Lady falling on the . wednesday the xxiiij daye of Marche. in the xix yeer of Kyng Edwarde the fourthe [1479]] 19193 xxxx Oct xx Various 1938-05-38 1963 to 1953 [not after 1579] 163[5?] muriel.foulonneau@tudor.lu 62 <a id="p63"></a> Everyone has a different definition of concepts Dimension (width x height) 48x48 64x64 Descriptive name very small Small 96x96 128x128 144x144 Medium Large extra large 160x160 super large 192x192 ridiculous large Jens Finke's reference sizes for thumbnails muriel.foulonneau@tudor.lu <a id="p64"></a> It is not clear to data providers what services will do with their data Anywhere? Title Creator/Contrib. Subject Type Creator Date muriel.foulonneau@tudor.lu <a id="p65"></a> Evolution of data sources By collection + regular increase 1 By collection only Regular increase No change Dead 0 1 2 3 4 Number of repositories 5 6 7 8 muriel.foulonneau@tudor.lu <a id="p66"></a> How fresh is the data? Broken links 404 muriel.foulonneau@tudor.lu 66 <a id="p67"></a> Data analysis based on service functions Impact of data characteristics on digital library functions • • Find (FRBR) Select (FRBR) Test on 176 collections (> 500.000 records) and 5.192 unique queries http://name.umdl.umich.edu/IC-KDIMG-X BBN7481-UND-0002-UND-001-UND 00000470]BBN7481_0002_001_00000470 Knight's American Mechanical Dictionary jpeg These pages may be freely searched and displayed. Permission must be received for subsequent electronically. distribution Please in print go or to http://www.umdl.umich.edu/ for more information. Knight's American mechanical dictionary image Industrial arts, Dictionaries, Mechanical engineering, Technology, Inventions 2; Knight's American mechanical dictionary : being a description of; tools, instruments, machines, processes, and engineering : history of; inventions : general technological vocabulary : and digest of mechanical; appliances in science and the arts; 3 v., 75 leaves of plates; Illustrated with upwards of five thousand engravings. UND Knight, Edward Henry lib.umich.edu.brutbib digital.lib.umn.edu.gpgovman lib.umich.edu.borobudurbib lib.umich.edu.ppotpusbib lib.umich.edu.cjsbib digital.lib.umn.edu.shm lib.umich.edu.postidbib etd.ohiolink.edu.osu iubio.bio.indiana.edu.biosoft.OAI2.iubio scout.wisc.edu.scout Average (mean) in the CIC repository muriel.foulonneau@tudor.lu % queries with 0 10% items retrieved 0 0 0 0.3 0.75 0.85 0.88 0.99 0.98 0.96 0.82 http://name.umdl.umich.edu/IC-KDIMG-X BBN7481-UND-0003-UND-001-UND 00000035]BBN7481_0003_001_00000035 Knight's American Mechanical Dictionary jpeg These pages may be freely searched and displayed. Permission must be received for subsequent electronically. distribution Please in print go or to http://www.umdl.umich.edu/ for more information. Knight's American mechanical dictionary image Industrial arts, Dictionaries, Mechanical engineering, Technology, Inventions 3; Knight's American mechanical dictionary : being a description of; tools, instruments, machines, processes, and engineering : history of; inventions : general technological vocabulary : and digest of mechanical; appliances in science and the arts; 3 v., 75 leaves of plates; Illustrated with upwards of five thousand engravings. UND Knight, Edward Henry % queries with =100% items retrieved 1 1 1 0.48 0.09 0.06 0.1 0 0 0 0.13 Affinity for retrieval 1 1 1 0.66 0.46 0.1 0.12 0.01 0.01 0.02 0.16 Resemblance of full record 0.97 0.86 0.82 0.51 0.31 0.26 0.19 0.04 0.03 0.01 0.39 67 <a id="p68"></a> Defining services Extension of IFLA FRBR specification (Functional Requirements for Bibliographic Records) user tasks • • • • Identify Obtain Find Select 29/04/2020 68 <a id="p69"></a> Defining the audience of each service and the corresponding data types User task Portal services Description Information in data Audience Find Find Locates a known item Locates unknown items Through [full] text indexes Pre-coordinated clustering Select Group Ranks Full text of record or selected fields Search engine [natural language] Terminologies (categories) Ranks the items located by relevance Full text Machine [application] Search engine [natural language] Select Organize Displays information to allow users assessing a resource (snippet) Organizes resources results or Selected fields and thumbnails / thumbshots Order resources by Title or Date Human Machine [application] Obtain [Relate] 29/04/2020 Manipulate Obtain Navigate Interpret (Coyle 2004 Describe) Interpret/Describe Groups / Clusters built out of pre-coordinated clustering Obtains the item Filters results collection by Type URL or ordering information Reach other resources not necessarily matching a query Allows users interpreting the resource by providing related contextual information Cluster collection) Select fields appearing before the identifier resource JPG resource) or Machine [application] (eg. allowing interpreting it (eg. the title of a Machine [application] Machine [application] Human 69 <a id="p70"></a> Aligning data and service functions Digital library Task or Function Select /Interpret Obtain Find / Rank Group Navigate Manipulate Organize Most common Dublin Core properties in the following order Title, Subject, Description, Creator, Identifier, Date, Publisher, Type, Format, Rights, Language, Contributor Source, Identifier Title, Creator, Subject, Description as Full records target fields Type, Date, Subject Type, Subject, Creator, Date Date, Type, Subject Title, Date, Creator, Subject 29/04/2020 70 <a id="p71"></a> Defining data requirement for each service Digital library system (DLS) service Find Audience type Search engine [natural language] Select Obtain Interpret Human Machine [application] Human Group/ Manipulate (Cluster) Navigate Machine [application] Machine [application] Organize 29/04/2020 Machine [application] Metadata syntax qualities Match natural language queries Does not generate a lot of noise Differentiates a record from others Explicit about the resource information about the resource) (bears Human friendly Differentiates a record from others Provides enough information to actually obtain the resource Human friendly Complete to provide as much context as possible on the resource Controlled terminology / list of keywords Controlled terminology or valid resolvable reference to related resources and Allows meaningful ordering 71 <a id="p72"></a> Data usability analysis for service Contributor Coverage Creator Date Description Format Identifier Language Publisher Relation Rights Source Subject Title Type 29/04/2020 # collection having property at least once # collections having property all the time Average length of property (bytes) 36 17 139 146 160 160 175 135 136 58 158 61 144 166 159 11 4 44 77 107 147 173 118 106 54 148 43 82 145 145 126.30 79.17 53.39 18.70 344.70 17.85 125.77 8.55 96.77 235.72 410.84 167.22 171.40 118.46 16.20 72 <a id="p73"></a> <h3>Evaluation of service creation experience (aggregation – based portals)</h3> muriel.foulonneau@tudor.lu 73 <a id="p74"></a> On the + side - Share content: New services, new representations, new audiences • 37% of visits to the picture collection of the State Library of New South Wales came from the Picture Australia portal in 2002/3 (now Trove) - Data synchronization muriel.foulonneau@tudor.lu <a id="p75"></a> On the side Need a critical mass of content Technical problems for data collection (XML formatting, data source availability …) Data quality Data traceability framework not well implemented No quasi automatic process Need communication between data and service providers The Entire Web Useful K-12 Educational Content 29/04/2020 75 <a id="p76"></a> <h3>Example 2- From data to service : semantic data sets</h3> 29/04/2020 Presentation Tudor 76 <a id="p77"></a> “To Really Learn, Quit Studying and Take a Test” (NYT, Jan 21, 2011) Formative assessment Self-assessment Items are expensive Creating, reusing, sharing test items 05/2011 ESWC 2011 77 <a id="p78"></a> Large item banks 29/04/2020 http://www.tao.lu/sites/tao/files/files/Publications/Plichart_ENAC2008-Berlin.pdf Presentation Tudor 78 <a id="p79"></a> Why generating items?  Large item banks for CAT (adaptive testing)  Cost issue  $1500-2000 per item  $3-4M estimated cost for the item creation only (Gierl et al., 2012).  Security issue  Adding variability to an item  Personalization of learning paths, formative assessment items, and on-demand assessment, informal learning environments  Scalability issue  Impossible that professionals create test items for all types of learners and contexts 29/04/2020 Presentation Tudor 79 <a id="p80"></a> 29/04/2020 <h3>What is assessment item generation?</h3> Presentation Tudor 80 <a id="p81"></a> Assessment I tem Generation, the w ay forward high cost of assessment item authoring and to create items adapted to the candidates or their context. Items can be generated from an item template, which includes variables in the stem, options, and auxiliary information (Gierl & Lai, 2013) AIG already demonstrated its value for instance in language learning (e.g., Brown, Frishkoff, & Eskenazi, 2005). However, it is used in a limited number of domains, where the generation of variables is the least challenging (Lai, Alves, & Gierl, 2009). This reduces the ability of AIG technologies to become mainstream and accompany the development of innovative forms of assessment, such as CAT and formative assessment. In order to support a wider range of application contexts, researchers need to cope with some of the critical limitations of the current approaches. In this paper, we aim to present some of the challenges faced by AIG and suggest orientations for future research. The Assessment Item Generation process The AIG process includes all steps from the conception of the item templates to the creation of items. Luecht (2013) propose an engineering process. Construct maps describe the skill and knowledge attached to different performance levels. They reflect the various proficiency claims on a score scale. A task model defines a combination of cognitive skills (i.e., procedural skills) applied to declarative knowledge objects required to support proficiency The AIG process instructions that provides skill and knowledge requirements to be reflected by an item AIG Evaluation Figure 1. AIG process 1) the definition of variables in an assessment task, Quality In other approaches, multiple items are generated from a particular learning resource, 2) the generation of items based on those variables, 3) statistical models used to estimate the psychometric properties of each item. between 1 and 10). Karamanis, Ha, & Mitkov (2006) generate questions from texts and Foulonneau (2012) and Papasalouros, Kotis, & Kanaris (2010) from semantic models. Zoumpatianos, Papasalouros, & Kotis (2011) even suggest generating choice questions from rules represented in SWRL (Semantic Web Rule Language). The variety of types of items and variables has led to the implementation of diverse processes. Recent efforts to standardize the process (Luecht, 2013) will help ensure the Difficulty Discrimination Quality models for generated items The processes must include quality assessment to compare the generated items to manually created items and to ensure the validity of the items, typically through calibration. Gierl, Zhou, & Alves (2008) proposed a model for assessing the quality of the generated items. They acknowledge however that a task model rarely exists for the item template to Page 2 <a id="p82"></a> The definition of variables in an item template options stem variables auxiliary information key 29/04/2020 Item extracted from Gierl, M.J., Zhou, J., Alves, C. (2008). Developing a Taxonomy of Item Model Types to Promote Assessment Engineering. Journal of Technology, Learning, and Assessment, 7(2). ESWC 2011 82 <a id="p83"></a> Existing strategies for the creation of variables • Algorithms X: Value range: 3 to 18 by 3 • Natural language processing Vocabulary questions and cloze questions • Structured datasets Vocabulary questions from the WordNet dataset • Domain model From natural language or creation of model by experts  Mostly used in mathematics and scientific subjects where algorithmic definition of variables is easier  And for L2 learning Challenge to generate other types of variables • Additional information, historical knowledge , feedback… 29/04/2020 83 <a id="p84"></a> Sources for the generation of items Texts Models Brown, J. C., Frishkoff, G. A., & Eskenazi, M. (2005). Automatic question generation for vocabulary assessment. In Processing Linguistics. Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language (pp. 819–826). Stroudsburg, PA, USA: Association for Computational 29/04/2020 Presentation Tudor 84 <a id="p85"></a> NLP based mapping to question types 29/04/2020 Shah (2012) 85 <a id="p86"></a> Models to represent knowledge or the organisation of topics Salles, P., Gontijo de Sá, I., Marques, G.F., Costa e Silva, P.A., Souza, A., Wilhems, L.H. and Resende, M.M.P. (2010). University of Brasília: Basic Topics and Models. DynaLearn, EC FP7 STREP project 231526, Deliverable D6.2.1. http://hcs.science.uva.nl/projects/DynaLearn/DeliverablesPublic/D6.2.1.pdf 29/04/2020 86 <a id="p87"></a> A set of semantic tools from models to items Semantic models Assessment item generator Test delivery platform 87 29/04/2020 Presentation Tudor <a id="p88"></a> <h3>Experimentation of item generation from Linked open Data</h3> 29/04/2020 Presentation Tudor 88 <a id="p89"></a> Using LoD for model-based learning Limitations of model-based learning  Experts are difficult to mobilize for a long modeling exercise LoD as a source of knowledge  Existing data sources No need to gather experts  What about specialized /professional knowledge?  Including knowledge which is not well codified in curricula Knowledge gathered from experts as well as non experts  How to ensure the evolution of the model?  Many datasets added or modified all the time Can reflect the evolution of the knowledge <a id="p90"></a> Objectives of the experimentation  Are there limitations to the use of Linked open Data as knowledge model for learning ? • • • Is this feasible? Are the datasets relevant? How much quality control is needed? Test on factual knowledge for choice items 29/04/2020 Presentation Tudor 90 <a id="p91"></a> The item generation process 29/04/2020 Presentation Tudor 91 <a id="p92"></a> The generation of items from Semantic Web data sources DBpedia Sindice OpenLink Also from local models (educational model in the health sector) Experimentation with the Dentistry department of the Karolinska Institute in Sweden 29/04/2020 Presentation Tudor 92 <a id="p93"></a> QTI Item template IMS Question & Test Interoperability Specification « The IMS Question & Test Interoperability (QTI) specification enables the exchange of item, test and results data between authoring tools, item banks, test constructional tools, learning systems and assessment delivery systems. » http://www.imsglobal.org/question/ XML serialization using JSON templates <choiceInteraction responseIdentifier="RESPONSE" shuffle="false" maxChoices="1"> <prompt>What is the capital of {prompt}?</prompt> <simpleChoice identifier="{responseCode1}">{responseOption1}</simpleChoice> <simpleChoice identifier="{responseCode2}">{responseOption2}</simpleChoice> <simpleChoice identifier="{responseCode3}">{responseOption3}</simpleChoice> </choiceInteraction> 29/04/2020 ESWC 2011 93 <a id="p94"></a> Get the knowledge from LoD Label SELECT ?country ?capital WHERE { ?c <http://dbpedia.org/property/commonName> ?country . ?c <http://dbpedia.org/property/capital> ?capital } LIMIT 30 SPARQL query to generate capitals in Europe Random selection of distractors among siblings 29/04/2020 ESWC 2011 94 <a id="p95"></a> Generation of the QTI-XML item 29/04/2020 ESWC 2011 95 <a id="p96"></a> Publication on the TAO e-assessment platform 29/04/2020 Presentation Tudor 96 <a id="p97"></a> Test with DBpedia data Who succeeded to {Charles VII the Victorious} as ruler of France ?  YAGO ontology  1 was inaccurate (The three Musketeers)  Multiple labels for the same king: Louis IX, Saint Louis, Saint Louis IX  One item generated with options having inconsistent naming: Charles VII the Victorious, Charles 09 Of France, Louis VII 29/04/2020 ESWC 2011 97 <a id="p98"></a> Data related issues • • • • • • Underlying ontologies Lack of persistence Transitive closures Reliability of data source Syntactic consistency of labels Data linkage Chance that an item will have a defective distractor = 29/04/2020 98 <a id="p99"></a> <h3>TO DO Find the mistake</h3> 29/04/2020 Presentation Tudor 99 <a id="p100"></a> <h3>TO DO Find the mistake</h3> 29/04/2020 Presentation Tudor 100 <a id="p101"></a> 29/04/2020 <h3>Selecting the data from the Semantic graph</h3> <h3>What is an educational domain model?</h3> 101 <a id="p102"></a> Various structures Taxonomy Model including influences http://hcs.science.uva.nl/projects/DynaLearn/DeliverablesPublic/D6.4.4.pdf 29/04/2020 Presentation Tudor 102 <a id="p103"></a> Types of relations in manually crafted educational models E.g., as found in the DynaLearn project Quantity spaces Factor influences + and – Positive and negative causal relations Conditional statements Representations of evolutions Zitek, A., Poppe, M., Stelzhammer, M., Jung, A., Zacharias, M., & Muhar, S. (2010). University of Natural Resources and Life Sciences, Vienna (BOKU) – Basic Topics and Models . DynaLearn, EC FP7 STREP project 231526, Deliverable D6.2.5. 29/04/2020 Presentation Tudor 103 <a id="p104"></a> Data transformation and enrichment Reusing Semantic Web sources as educational model But what is missing? 1. Boundaries of the model: width and depth 2. 3. Audience Mechanism to define a learning path 4. 5. Mechanisms to use the model Define the useful types of properties for particular learning and assessment purposes (e.g., causal properties, process-related properties) 29/04/2020 104 <a id="p105"></a> Mapping Knowledge to model patterns  Factual knowledge : e.g., holonymy , and meronymy  Conceptual knowledge e.g., hypernymy , hyponymy , and cause/effect relations  Procedural knowledge e.g., formula , procedure , and condition relations Knowledge dimension of Taxonomy of Bloom (Ying & Yang, 2010) Presentation Tudor 105 29/04/2020 <a id="p106"></a> Model based mapping to question types (different typologies) 29/04/2020 Linnebank, F., Liem, J., and Bredeweg, B. (2010). Question generation and answering . DynaLearn, EC FP7 STREP project 231526, Deliverable D3.3. Presentation Tudor 106 <a id="p107"></a> <h3>Specific data selection processes The generation of distractors (erroneous options)</h3> 29/04/2020 Presentation Tudor 107 <a id="p108"></a> What is a “good” distractor? “a concept semantically close to the key which, however, cannot serve as the right answer itself ” (Shah, 2012) - For a choice item , good distractors are not overlapping but sufficiently close to the correct answer to decrease the odds that the test takers will find them by chance 29/04/2020 Presentation Tudor 108 <a id="p109"></a> Semantic similarity metrics Taxonomy similarity: similarity based on the comparison of all the concepts to which each instance belongs. Relation similarity: similarity relative to the common relations of the two instances with other objects. This includes incoming and outgoing properties. Attribute similarity: similarity of the two instances regarding to their attributes (comparison of the literal values of their datatype properties). sim ( I i , I j ) = t ´ TS ( I i , I j ) + r ´ t RS ( I i , I + r + a j ) + a ´ AS ( I i , I j ) 109 <a id="p110"></a> 29/04/2020 <h3>TO DO What is the capital of Gabon?</h3> Presentation Tudor 110 <a id="p111"></a> <h3>TO DO What would be good distractors for this question?</h3> 29/04/2020 Presentation Tudor 111 <a id="p112"></a> Calculating the semantic similarity (graph based) between distractors and the correct answer What is the capital of Gabon? Ulan Bator Libreville Manila No SemSim Maputo Port Louis Libreville With SemSim 29/04/2020 112 <a id="p113"></a> Results of user test Clear decrease of performance in the population when using SemSim 29/04/2020 113 <a id="p114"></a> <h3>Evaluation of service creation experience (Web domain model for education)</h3> muriel.foulonneau@tudor.lu 114 <a id="p115"></a> On the + side - Dynamic domain models Semantic descriptions of concepts Data linkages muriel.foulonneau@tudor.lu <a id="p116"></a> On the side - Reliance of third party concepts Difficulty to query multiple data sources: reliance on search engines / data aggregators Data quality Significant manual reprocessing 29/04/2020 116 <a id="p117"></a> 29/04/2020 <h3>Example 3 From data to service : Mining the Web </h3> <h3>Webpages, paradata, social networks</h3> Presentation Tudor 117 <a id="p118"></a> - Identifying common vs. expert knowledge from Web mining Based on Google estimation of responses Closer shape when adding language restriction The stem variable popularity was a better indicator than the answer variable Need to connect knowledge to a domain rather than isolated facts ? Search on Google with the language of test takers Angola 45900000 Luanda Bangladesh Bénin Botswana 32700000 Dacca 49300000 Porto-Novo 25300000 Gaborone 2060000 559000 2650000 1900000 29/04/2020 Brunei Burkina Faso Cameroun Centrafrique Gabon Lesotho 20100000 Bandar Seri Begawan 42000000 Ouagadougou 66200000 Yaoundé 5110000 Bangui 36200000 Libreville 43800000 Maseru 4900000 11000000 9200000 1350000 3480000 2020000 118 <a id="p119"></a> Using correlations between user queries (paradata) and life events Find: -Strong correlation bt: search keywords & unemployment rates -Internet activity help predict complex and fast changing conditions -Sources of data on real time economic activity Search engines forecast: Automobiles sales, unemployment claims, travel destinations, consumers confidence <a id="p120"></a> Data sampling - Google not a good sensor for all countries - Google happens to provide good results NOW. Need to keep measuring the effectiveness of the sensor to be able to use it - Who are the Google users vs. the Bing users? Will that change in the near future? - Geographic data, demographic data etc are often missing <a id="p121"></a> Personalization of the Web Pb= correlation with existing indexes - BUT Dynamic pricing mechanisms: no 2 people will have the same price Prices adapted to the number of times I accessed a service with a request Access to different trips on expedia.fr and expedia.be but I can use .be even if I live in France (biased sampling) 29/04/2020 121 <a id="p122"></a> Data source reliability and timeseries The Web has no (built in) memory - Can we recreate a posteriori timeseries? Web archives www. data . gov / [14/10/2013] How can we validate the time of a first data collection mechanism? - Timestamps for social data - Regular data harvesting for Web data 29/04/2020 122 <a id="p123"></a> <h3>Summary on Data based on example services</h3> muriel.foulonneau@tudor.lu 123 <a id="p124"></a> From data to services Various data access mechanisms (OAI-PMH, LoD, data dumps, wrappers, ….) Whichever the type of data and the access mechanism, similar issues => Data reusability Data producers make their data available for reuse BUT Reuse does not always happen => Need to understand the service creation process 29/04/2020 124 <a id="p125"></a> 29/04/2020 <h3>Part III- Designing services from data</h3> Presentation Tudor 125 <a id="p126"></a> Service design Investigation Topic Idea generation Maturation Concept evaluation The Service Brick Road: a tool supporting rapid service protoyping, 2013, Johannsen L. , Vidou G, XXIV IPISM Conference, June 2013, Helsinki 29/04/2020 126 <a id="p127"></a> Service and data: which relationship? Services & data: - Services based on data / data as impulses for the service - Services with data as resources - Services validated or enriched with data / data as enrichment or validation for other data 29/04/2020 Presentation Tudor 127 <a id="p128"></a> 29/04/2020 <h3>Case 1 – Services based on data</h3> Presentation Tudor 128 <a id="p129"></a> Ideation phase: data as impulses 1- Who uses these data ? 2- Who else could be interested by these data ? 3- What about combination of these data with other data ? 29/04/2020 129 <a id="p130"></a> New meaning to data: Open Data 29/04/2020 130 <a id="p131"></a> New meaning to data: Google ngrams 29/04/2020 131 <a id="p132"></a> Maturation phase: data as resources - Checklist about data: Updates Quality (reliability, completeness, etc) Origin Maintenance processes IP rights & conditions of use Costs Accessibility: technical (API, dump), format (RDF/XML, JSON …) Interoperability: semantic models Documentation 29/04/2020 132 <a id="p133"></a> 29/04/2020 <h3>Case 2 – Services with data as resources</h3> Presentation Tudor 133 <a id="p134"></a> Maturation phase - Questions related to services maturation: updates, IP rights, Etc Ex. transport and traffic data for delivery service 29/04/2020 134 <a id="p135"></a> <h3>Case 3 – Data as enrichment or validation of services or its resources</h3> 29/04/2020 Presentation Tudor 135 <a id="p136"></a> Data enrich or validate other data Ex. Use of Wikipedia for concepts translation Ex. Postal code validation Ex. Use of WordNet for analysis of text cohesion in an evaluation service of texts readability 29/04/2020 http://www.cs.princeton.edu/courses/archive/spr07/cos226/assignments /wordnet.html http://www.cs.princeton.edu/courses/archive/spr13/cos226/assignments /wordnet.html 136 <a id="p137"></a> Concept validation Service validation: ex. recommendation system Business model validation 29/04/2020 137 <a id="p138"></a> Mix of licenses Several strategies - Europeana pour CC0 - Portail Open Data of Singapore for documentation of IP conditions 29/04/2020 138 <a id="p139"></a> Services and applications designed from data • • • Design 1 service with: 1 dataset 2 and more datasets a mashup with licensed data Use of data • Only to enrich existing data • Only to improve quality of existing data (ex. data accuracy) 29/04/2020 139 <a id="p140"></a> <h3>TO DO Can you name services/applications developed on data</h3> 29/04/2020 Presentation Tudor 140 <a id="p141"></a> Narrative science « Bridging the Gap between Numbers and Knowledge » http://www.nytimes.com/2011/09/11/business/computer-generated articles-are-gaining-traction.html?_r=2&ref=start-ups& 29/04/2020 141 <a id="p142"></a> Europeana hackhatons http://paths.sheffield.ac.uk/storyana 29/04/2020 142 <a id="p143"></a> Public Spending .net 29/04/2020 143 <a id="p144"></a> Application to support transparency and democracy nosdeputes.net 29/04/2020 144 <a id="p145"></a> Salaries of government employees “Source: Salary data provided by agencies under the Texas Public Information Act | Download a spreadsheet of our salary data ( Terms )” 29/04/2020 145 <a id="p146"></a> Citizen services Paths for mobility of disabled in the city (Montpellier and Rennes in France) 29/04/2020 146 <a id="p147"></a> Game to discover the city of London 29/04/2020 147 <a id="p148"></a> Companies created on data based services E.g., Enigma.io 29/04/2020 148 <a id="p149"></a> Public data visualization 29/04/2020 149 <a id="p150"></a> Facilitate creation of services APIs Platforms: Callimachus 29/04/2020 150 <a id="p151"></a> Which services? Data.gov.uk : 301 apps Open data Singapore : > 170 apps - Economy: Property prize , workmarket Environment: Use of renewable energies Transparency: Availability of governmental financial data « Where does my money go? » Society: Criminality map Useful services: Best surgery services next to me, Postal code where I am, Fuel station search Education: geological layers where I stand Citizen life : elections results , « UK arms export licence browser » 29/04/2020 151 <a id="p152"></a> Models of data reuse - On 106 UK applications, use of datasets: 83/106 = 1 dataset - Sketchmap.co.uk uses 19 datasets : wonderful ! - Most used datasets : Code Point® Open, National Public Transport Access Nodes (NaPTAN) - => applications clones (ex. postal codes) => success of transport datasets reuse => to notice : NaPTAN is available in RDF - 184/231 free applications, others most often <1£ - Most downloaded app: via Google Play, NHS direct app (medical diagnostic) > 100.000 - => an administration reuses its own data 29/04/2020 152 <a id="p153"></a> Used formats 29/04/2020 http://www.slideshare.net/ghislainatemezing/visualisation-and-linked data-applications-edf-2013 153 <a id="p154"></a> 29/04/2020 <h3>TO DO How to encourage reusers?</h3> Presentation Tudor 154 <a id="p155"></a> Build an innovation strategy on open datasets Automatically include conditions of use of data Availbility of API in order to reduce the service development cost 29/04/2020 155 <a id="p156"></a> A service-oriented Open Data platform Contests : hackhatons, service ideas  Advertising on datasets Call-For-Collaborations: identify high-potential services and participate in their development  Creation of services 29/04/2020 http://www.biginnovationcentre.com/Assets/Docs/Reports/Hackday%2 0Lessons%20Learnt_final.pdf 156 <a id="p157"></a> Competitions with public and private partnerships 29/04/2020 157 <a id="p158"></a> Open Government Data strategy in Germany - Showroom - Support service to re-users Choice of datasets to open with re-users (see also the French ministry of culture call for interest in cultural data openness) - Discussion via social networks http://www.bmi.bund.de/SharedDocs/Downloads/DE/Themen/OED_Verwaltung/ModerneVerwaltung/opengovernment.pdf ?__blob=publicationFile 29/04/2020 158 <a id="p159"></a> Education application : data selection 29/04/2020 159 <a id="p160"></a> Competitions 29/04/2020 160 <a id="p161"></a> Community building 29/04/2020 161 <a id="p162"></a> Link our own data : infrastructure exemples 29/04/2020 http://stefandietze.files.wordpress.com/2011/05/20110529_linkedlearni ng2_eswc_fouad_zablith_etal.pdf 162 <a id="p163"></a> Examples of applications 29/04/2020 http://stefandietze.files.wordpress.com/2011/05/20110529_linkedlearni ng2_eswc_fouad_zablith_etal.pdf 163 <a id="p164"></a> <h3>TO DO Which datasets are most popular?</h3> 29/04/2020 Presentation Tudor 164 <a id="p165"></a> Analysis of services based on Open Data 45 40 35 30 25 20 15 10 5 0 Thematic distribution of applications based on datasets published on data.gov.uk Keywords related to apps (2011) 29/04/2020 http://digit4cities.files.wordpress.com/2011/05/louverture des-donnc3a9es-publiques-au-royaume-uni.pdf 165 <a id="p166"></a> Potential of datasets for service design Designed services vs. Available datasets 1000 900 800 700 600 500 400 300 200 100 0 29/04/2020 166 <a id="p167"></a> Interest of the public 29/04/2020 http://opendata.paris.fr/opendata/ 167 <a id="p168"></a> <h3>TO DO Imagine a service based on data of the Neumunster Abbey</h3> 29/04/2020 Presentation Tudor 168 <a id="p169"></a> <h3>Summary on Services</h3> muriel.foulonneau@tudor.lu 169 <a id="p170"></a> From service concept to data Multiple types of services Multiple types of service creators Various roles of data in services Risks and opportunities of data in service design Open or closed datasets Dynamic data New role of data in today’s economy 29/04/2020 170 <a id="p171"></a> Services based on data Demographic change Economic change data technology 29/04/2020 Weather change ? 171 <a id="p172"></a> Some references Muriel Foulonneau, Valentin Grouès. Common vs. Expert knowledge: making the Semantic Web an educational model. In Proceedings of the 2nd International Workshop on Learning and Education with the Web of Data (LiLe-2012 at WWW-2012). Stefan Dietze, Mathieu d'Aquin, Dragan Gasevic (eds.). CEUR workshop proceedings, vol. 840. http://www.ceur-ws.org/Vol-840/ Muriel Foulonneau. Generating Educational Assessment Items from Linked Open Data: the Case of DBpedia. In R. Garcia-Castro et al. (Eds.): ESWC 2011 Workshops, LNCS 7117, pp. 16--27. Springer, Heidelberg (2011) Foulonneau, Muriel, Cole, Timothy W. . Strategies for reprocessing aggregated metadata Lecture notes in computer science 3652 , 290-301 (2005) Muriel Foulonneau , “Information redundancy across metadata collections,” Information Processing & Management, May 2007, Timothy W. Cole and Muriel Foulonneau "Using the Open Archives Initiative Protocol for Metadata Harvesting", Greenwood Publishing, 2007, US Muriel Foulonneau, Jenn Riley, "Metadata for Digital Resources – Implementation, systems design and interoperability" Chandos Publishing, 2008, UK 29/04/2020 172 <a id="p173"></a> More references H.W. Chesbrough, and M.M. Appleyard , “Open innovation and strategy”. California Management Review, vol. 50, pp. 57-76, 2007. Martin, S., Foulonneau, M., Turki, S. (2013). 1-5 stars: Metadata on the Openness Level of Open Data Sets in Europe. In Metadata and Semantics Research 2013. Communications in Computer and Information Science, Springer Verlag, Heidelberg Foulonneau, M., Martin, S., Ducloy, J., Daunois, T., Turki, S. (2013). Usage data for metadata properties to support open data registries and semantic wikis. CAMP4Data 2013, workshop co-located with the Dublin Core Conference, Lisbon, Portugal Muriel Foulonneau, Eric Ras. (2013). Using Educational Domain Models for Automatic Item Generation Beyond Factual Knowledge Assessment. EC-TEL conference 2013 Sébastien Martin, Muriel Foulonneau, Slim Turki, Majid Ihadjadene (2013). Open Data: Barriers, Risks, and Opportunities. European Conference on eGovernment, Como, Italy, 13-14th June, 2013. 29/04/2020 173 <a id="p174"></a> Even more references Ilkka Tuomi, Data is More Than Knowledge - Implications of the Reversed Knowledge Hierarchy for Knowledge Management and Organizational Memory, Journal of Management Information Systems / Fall 1999, Vol. 16, No. 3., pp 107 121 Ackoff, R. L., "From Data to Wisdom", Journal of Applies Systems Analysis, Volume 16, 1989 p 3-9. Milan Zeleny, "Management Support Systems: Towards Integrated Knowledge Management," Human Systems Management 7, no 1 (1987): 59-70. Retrieved July 7 th , 2010 from http://www.milanzeleny.com/documents/publications/mss.pdf Adam Maria Gadomski, Meta-Ontological Assumptions: Information, Preferences and Knowledge universal interrelations, 1999 (cognitive IPK architecture) [1] Boiko, Defining Data, Information, and Content - A CM Domain White Paper, first chapter of the book “The Content Management Bible”, Wiley and Son, 2001. Retrieved on July 9, 2010 from http://www.metatorial.com/downloads/Boiko_Wp_DefiningDataInformationCont ent.pdf Paul Beynon-Davies, Neolithic informatics: The nature of information, International Journal of Information Management 29 (2009) 3 –14 29/04/2020 174 <a id="p175"></a> Other relevant references Gierl, M. J., & Haladyna, T. M. (2013). Automatic Item Generation: Theory and Practice. Routledge, New York. Ying, M.-H., & Yang, H.-L. (2008). Computer-Aided Generation of Item Banks Based on Ontology and Bloom’s Taxonomy. In F. Li, J. Zhao, T. K. Shih, R. Lau, Q. Li, & D. McLeod (Eds.), Advances in Web Based Learning - ICWL 2008 (pp. 157 –166). Springer Berlin Heidelberg. Gierl, M.J., Zhou, J., Alves, C. (2008). Developing a Taxonomy of Item Model Types to Promote Assessment Engineering. Journal of Technology, Learning, and Assessment, 7(2). Sha, R. (2012). Automatic question generation using discourse cues and distractor selection for cloze questions. Master thesis. Language technology and research center (LTRC), International Institute of Information Technology, Hyderabad, India. </div> </section> </div> </div> </div> </main> <footer> <div class="container mt-3"> <div class="row justify-content-between"> <div class="col"> <a href="/"> <img src="/theme/studyslide/static/logo-slideum.png" /> </a> </div> </div> <div class="row mt-3"> <ul class="col-sm-6 list-unstyled"> <li> <h6 class="mb-3">Company</h6> <li> Nicosia Constantinou Palaiologou 16, Palouriotissa, 1040 <li> +357 64-733-402 <li> info@slideum.com </ul> <ul class="col-6 col-sm-3 list-unstyled"> <li> <h6 class="mb-3">Links</h6> <li> <a href="/about">About</a> <li> <a href="/contacts">Contact</a> <li> <a href="/faq">Help / FAQ</a> </ul> <ul class="col-6 col-sm-3 list-unstyled"> <li> <h6 class="mb-3">Legal</h6> <li> <a href="/terms">Terms of Service</a> <li> <a href="/privacy">Privacy policy</a> <li> <a href="/page.html?code=public.usefull.cookie">Cookie policy</a> <li> <a href="/page.html?code=public.usefull.disclaimer">Disclaimer</a> </ul> </div> <hr> slideum.com © 2026, Inc. All rights reserved. </div> </footer> <div class="modal directory" id="directory-modal"> <div class="modal-dialog"> <div class="modal-content"> <div class="modal-header"> <h5 class="modal-title">Directory</h5> <button class="close" type="button" data-dismiss="modal">×</button> </div> <div class="modal-body"></div> </div> </div> </div> <script src="/theme/common/static/jquery@3.6.0/dist/jquery.min.js"></script> <script src="/theme/common/static/jquery_extra/dist/jquery-extra.js"></script> <script src="/theme/common/static/popper.js@1.16.1/dist/umd/popper.min.js"></script> <script src="/theme/common/static/bootstrap@4.6.0/dist/js/bootstrap.min.js"></script> <script> var __path_directory = [ ] !function __draw_directory(data, root, uuid) { var ul = $('<ul>', uuid && { id: 'category' + uuid, class: !__path_directory.includes(uuid) ? 'collapse' : null }); for (var item in data) { var li = $('<li>').appendTo(ul); if (item = data[item], item.children) { li.append('<a href=#category' + item.id + ' data-toggle=collapse>') __draw_directory(item.children, li, item.id); } else { li.append('<a href=' + item.url + '>'); } var a = $('> a', li).addClass('item').text(item.name) .append($('<a class="link fa fa-external-link" href=' + item.url + '>')); if (item.id === +__path_directory.slice(-1)) { a.addClass('active'); } /* if (item.id !== __path_directory[0]) { a.addClass('collapsed'); } */ } root.append(ul); } ([{"id":1,"name":"Food and cooking","url":"/catalog/Food+and+cooking","children":null},{"id":2,"name":"Education","url":"/catalog/Education","children":null},{"id":3,"name":"Healthcare","url":"/catalog/Healthcare","children":null},{"id":4,"name":"Real estate","url":"/catalog/Real+estate","children":null},{"id":5,"name":"Religion ","url":"/catalog/Religion+","children":null},{"id":6,"name":"Science and nature","url":"/catalog/Science+and+nature","children":null},{"id":7,"name":"Internet","url":"/catalog/Internet","children":null},{"id":8,"name":"Sport","url":"/catalog/Sport","children":null},{"id":9,"name":"Technical documentation","url":"/catalog/Technical+documentation","children":null},{"id":10,"name":"Travel","url":"/catalog/Travel","children":null},{"id":11,"name":"Art and Design","url":"/catalog/Art+and+Design","children":null},{"id":12,"name":"Automotive","url":"/catalog/Automotive","children":null},{"id":13,"name":"Business","url":"/catalog/Business","children":null},{"id":14,"name":"Government","url":"/catalog/Government","children":null}], $('#directory-aside')); var __root_directory = $('#directory-aside > ul'); $('#directory-aside') .on('show.bs.collapse', function() { //console.log('show.collapse') }) .on('hide.bs.collapse', function() { //console.log('hide.collapse') }); $('#directory-modal') .on('show.bs.modal', function() { $('[class$="body"]', this).prepend(__root_directory); }) .on('hide.bs.modal', function() { $('#directory-aside').prepend(__root_directory); }); $('.directory-mobile').on('click', function(e) { e.preventDefault(); }); $('.directory .link').on('click', function(e) { e.stopPropagation(); }); </script> <script> function scrollToViewport() { $('html, body').stop().animate( { scrollTop: $('.navbar').outerHeight() }, 1000); } setTimeout(scrollToViewport, 1000); $(window).on('orientationchange', scrollToViewport); $('[data-toggle="tooltip"]').tooltip(); </script> <script async src="//s7.addthis.com/js/300/addthis_widget.js#pubid=#sp('addthis_pub_id')"></script> <script type="text/javascript"> (function (d, w, c) { (w[c] = w[c] || []).push(function() { try { w.yaCounter28397281 = new Ya.Metrika({ id:28397281 }); } catch(e) { } }); var n = d.getElementsByTagName("script")[0], s = d.createElement("script"), f = function () { n.parentNode.insertBefore(s, n); }; s.type = "text/javascript"; s.async = true; s.src = (d.location.protocol == "https:" ? "https:" : "http:") + "//mc.yandex.ru/metrika/watch.js"; if (w.opera == "[object Opera]") { d.addEventListener("DOMContentLoaded", f, false); } else { f(); } })(document, window, "yandex_metrika_callbacks"); </script> <noscript><div><img src="//mc.yandex.ru/watch/28397281" style="position:absolute; left:-9999px;" alt="" /></div></noscript> <link rel="stylesheet" type="text/css" href="//cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.1.0/cookieconsent.min.css" /> <style> @media screen and (max-width: 768px) { .cc-revoke { display: none; }} </style> <script src="//cdnjs.cloudflare.com/ajax/libs/cookieconsent2/3.1.0/cookieconsent.min.js"></script> <script> window.addEventListener("load", function() { window.cookieconsent.initialise( { content: { href: "https://slideum.com/dmca" }, location: true, palette: { button: { background: "#fff", text: "#237afc" }, popup: { background: "#007bff" }, }, position: "bottom-right", revokable: true, theme: "classic", type: "opt-in" })}); </script> </body> </html>