Transcript Query Processing in Heterogeneous Systems

Information Integration
1
Information Resides on
Heterogeneous Information Sources
WWW
Flat File
Excel
Personal
database
• different interfaces
• different data representations
• redundant and conflicting information
2
Modes of Information Integration
• Federated Databases: the sources are independent,
but one source can call on others to supply information
• Data warehouse: copies of data from several sources
are stored in a single database, called a data warehouse.
The data stored at the warehouse is first processed in some
way before storage; e.g. data may be filtered, and relations
may be joined or aggregated. As the data is copied from
the sources, it may need to be transformed in certain ways
to make all data conform to the schema at the data
warehouse
3
Modes of Information Integration
• Mediation: a mediator is a software component that
supports a virtual database, which the user may query as if
it were materialized (physically constructed like a
warehouse). The mediator store no data of its own. Rather,
it translates the user’s query into one or more queries to its
sources. The mediator then synthesizes the answer to the
user’s query from the responses of those sources, and
returns the answer to the user
4
Problems of Information Integration
Example:
The AAAI Automobile Co. has 1000 dealers each of which
maintains a database of their cars in stock. AAAI wants to
create an integrated database containing the information of
all 1000 sources. The integrated database will help dealers
locate a particular model if they don’t have one in stock. It
also can be used by corporate analysts to predict the
market and adjust production to provide the model most
likely to sell
5
Problems of Information Integration
• The 1000 dealers do not all use the same database schema:
Cars (serialNo, model, color,
autoTrans, cdPlayer, ...)
or
Autos (serialNo, model, color)
Options (serialNo, option)
6
Problems of Information Integration
• Schema difference
• Different equivalent names
• Data type differences: numbers may be represented by
character strings of varying length at one source and fixed
length at another
• Value differences: the same concept may be represented
by different constants at different sources (BLACK, BL,
100, etc)
• Semantic differences: Terms can be given different
interpretations at different sources (Cars includes trucks
or not)
• Missing values: a source may not record information of a
type that all of the other sources provide
7
Goal: System Providing Integrated
View of Heterogeneous Data
Integration System
WWW
Flat File
Excel
• collects and combines information
• provides integrated view, uniform user
interface
Personal
database
8
The Data Warehousing Approach
to Integration
Client
Mediator
Wrapper
Excel
Stored
Integrated
View
Wrapper
Flat File
9
The Data Warehousing Approach to
Integration
• Data from several sources is extracted and combined
into a global schema
• The data is stored at the warehouse which looks like an
ordinary database
• There are three approaches to maintaining the data in
the data warehouse:
– off-line reconstruction of the whole data warehouse
– the data warehouse is updated periodically based on the
changes made to the original data sources
– the data warehouse is updated immediately
10
The Data Warehousing Approach to
Integration
Example
Suppose that there are two dealers in the system and
that they use the schemas:
Cars (serialNo, model,color,autoTrans,
cdPlayer, ...)
and
Autos (serialNo,model,color)
Options (serialNo,option)
Assume a data warehouse with the schema:
AutoWhse(serialNo,model,color,autoTrans,
dealer)
11
The Data Warehousing Approach to
Integration
• The software to extracts data from the dealer’s
databases and populates the global schema can be
written as SQL-queries. The query for the first dealer:
insert into AutoWhse(serialNo,model,color,autoTrans,
dealer)
select serialNo, model, color, autoTrans, ‘dealer1’
from Cars
The code for the second dealer is more complex since we
have to decide whether or not a given car has an
automatic transmission.
12
The Data Warehousing Approach to
Integration
insert into AutoWhse(serialNo,model,color,autoTrans,
dealer)
select serialNo, model, color, ‘yes’, ‘dealer2’
from Autos, Options
where Autos.serialNo=Options.serialNo and
option=‘autoTrans’;
insert into AutoWhse(serialNo,model,color,autoTrans,
dealer)
select serialNo, model, color, ‘no’ ‘dealer2’
from Autos
where not exists ( select * from Options
where serialNo=Autos.serialNo and
option=‘autoTrans’);
13
The Wrapper and Mediator
Architecture
Client
Common
Data
Model
portfolios for each company
Mediator
stock market prices
Wrapper
Excel
business reports
Wrapper
Flat File
14
The Wrapper and Mediator
Architecture
• A mediator supports a virtual view, or collection of
views, that integrates several sources in much the same
way that the materialized relation(s) in a data
warehouse integrate sources.
• The mediator doesn’t store any data <
<
<
Example:
Let us consider the same scenario. The mediator
integrates the same two data sources into a view that is
a single relation with the schema:
AutoMed(serialNo,model,color,autoTrans,
dealer)
15
The Wrapper and Mediator
Architecture
Assume the user asks the mediator about the red cars:
select serialNo, model from AutosMed
where color = ‘red’;
The mediator forward the same query to each of the
two wrappers
(1) select serialNo, model from Cars where color=‘red’;
(2) select serialNo, model from Autos where color=‘red’;
The mediator can take the union of answers and return
the result to the user.
16
The Lazy Integration Approach
Query Decomposition,
Translation and
Result Fusion
Client
IBM portfolio
Mediator
IBM price
Wrapper
IBM related reports
(in common model)
Wrapper
IBM related reports
Excel
Flat File
17
Wrappers in Mediator-Based Systems
• In a data warehouse system, the source extractors
consist of:
– one or more queries built-in that are executed at the source to
produce data for the data warehouse
– communication mechanisms, so that wrapper can:
• pass ad-hoc queries to the source
• receive responses from the source
• pass information to the warehouse
• Mediator systems require more complex wrappers - the
wrapper must be able to accept a variety of queries
from the mediator and translate any of them to the
terms of the source.
18
Wrappers in Mediator-Based Systems
• A systematic way to design a wrapper that connects a
mediator to a source is to classify the possible queries
that the mediator can ask into templates, which are
queries with parameters that represent constants.
• The mediator can provide the constants, and the
wrapper executes the query with the given constants.
• T  S the template T is turned into the source query S
Example:
The source of dealer1:
Cars (serialNo, model,color,autoTrans,
cdPlayer, ...)
19
Wrappers in Mediator-Based Systems
Assume we use the mediator with schema:
AutoMed(serialNo,model,color,autoTrans,
dealer)
How the mediator could ask the wrapper for cars of a
given color>
The template:
select * from AutoMed where color= ‘$c’; 
select serialNo, model color, autoTrans, ‘dealer1’
from Cars where color=‘$c’;
20
Wrappers in Mediator-Based Systems
• The wrapper could have another template that
specified the parameter $m representing a model
• there would be 2N templates for N attributes
• the number of templates could grow unreasonably
large.
21
Wrapper Generators
• The template defining a wrapper must be turned into
code for the wrapper itself - the software that creates
the wrapper is called a wrapper generator
• The wrapper generator creates a table that holds the
various query patterns contained in the templates, and
the source queries that are associated with each.
• A driver is used in each wrapper. The task of the driver
is to :
– accept a query from the mediator
– search the table for a template that match the query
– the source query is sent to the source using a communication
mechanism
– the response is processed by the wrapper, if necessary, and
then returned to the mediator
22
Wrappers & Mediators from
High-Level Specifications
Client
Mediator Specification
Interpreter
Mediator
Mediator
Specification
Wrapper
Source
Wrapper
Wrapper
Generator
Source
Wrapper
Specification
23
Filters
• Complex template
select * from AutoMed
where color= ‘$c’ and model = ‘$m’; 
select serialNo, model color, autoTrans, ‘dealer1’
from Cars where color=‘$c’ and model =‘$m’;
• Wrapper filter approach - if the wrapper has a
template that returns a superset of what the query
wants then it is possible to filter the result at the
wrapper
• The decision whether a mediator asks for a subset of
what the pattern of some wrapper template returns is a
hard problem
<
<
<
<>>>>
24
Filters
Example:
Given the template
select * from AutoMed where color= ‘$c’;
The mediator needs to find blue Gobi model car:
select * from AutoMed where color= ‘blue’ and
model=‘Gobi’;
• use the template with $c=blue to find all blue cars
• store the result in the temporary relation Temp
• select from Temp the Gobi’s and return the result
25
Other Wrapper Operations
• It is possible to transform the data at the wrapper in
different ways
• The mediator is asked to find dealers and models such
that the dealer has two red cars, of the same model, one
with and one without automatic transmission. Suppose
we have only one template as before.
Select A1.model A1.dealer
from AutoMed A1 AutoMed A2
where A1.model=A2.model and A1.color=‘red’ and
A2.color=‘red’ and A1.autoTrans=‘no’ and
A2.autoTrans=‘yes’;
26
Other Wrapper Operations
• It is possible to answer the query by first obtaining
from the Dealer’s 1 source a relation with all the red
cars (use the original template) - RedAutos relation
• select distinct A1.model A1.dealer
from RedAutos A1, RedAutos A2
where A1.model=A2.model and
A1.autoTrans=‘no’ and
A2.autoTrans=‘yes’;
27
Challenge: Sources Without a WellStructured Schema
• semistructured
– irregular
– deeply nested
– cross-referenced
• incomplete
schema knowledge
– autonomous
– dynamic
Examples
•
•
•
•
•
HTML pages
SGML documents
genome data
chemical structures
bibliographic
information
• results of the
integration process
28
Challenge: Different and Limited
Source Capabilities
Client
retrieve IBM data
Mediator
(U = A + B)
retrieve IBM data
Wrapper
(A)
retrieve IBM data
Wrapper
(B)
29
Mediator has to Adapt to Query
Capabilities of Sources
Client
retrieve IBM data
Mediator
(U = A + B)
retrieve IBM data
retrieve everything
(A) does not
allow selection
Wrapper
(A)
retrieve IBM data
Wrapper
(B)
30
Part B
•
•
•
•
Semistructured Data Representation
Mediator Generation
Wrapper Generation
Capabilities-Based Rewriting
31
Representation of Semistructured
Information using OEM
semantic
object-id
label
Set Value
<http://www/~doe, faculty, {&f1,&l1,&r1}>
<&f1, first_name, “John”>
Atomic Value
<&l1, last_name, “Doe”>
<&r1, rank, “professor”>
structural
object-id
32
Object Exchange Model - Goals
•
•
•
•
Easy to read
Easy to edit
Easy to generate or parse by a program
Consistency with Stanford’s other
projects (developed with the TSIMMIS’)
• Possibility of extensions in the future
33
Graph Representation of OEM
Data
<http://www/~doe, faculty, {&f1,&l1,&r1}>
<&f1, first_name, “John”>
<&l1, last_name, “Doe”>
<&r1, rank, “professor”>
http://www/~doe faculty
first_name “John”
last_name “Doe”
rank “professor”
34
OEM Structures Represent
Arbitrary Labeled Graphs
http://www/~smith
faculty
name “Mary Smith”
project “Air DB”
paper
author
name “John Doe”
author
name “Mary Smith”
title “Thin Air DB”
http://www/~doe
faculty
first_name “John”
last_name “Doe”
rank “professor”
35
Reprezentacja danych
semistrukturalnych
•
•
•
•
Object Exchange Model
ACeDB
XML
Mogą być wykorzystywane w warstwie
pomiędzy mediatorem a wrapper’ami.
36
Object Exchange Model
• Zdefiniowany przy okazji budowy
systemu Tsimmis służącego do integracji
heterogenicznych źródeł danych.
• Wykorzystywany przy projekcie Merlin
(MQS) i Lorel (QL)
37
Object Exchange Model (cd)
• Węzeł OEM skłąda się z czterech pól:
– Object-ID – jest wykorzystywany do unikalnej
identyfikacji określonego węzła OEM
– Label – jest ciągiem znaków który opisuje to co
węzeł OEM reprezentuje
– Type – jest typem danych wartości węzła. (atomowy
lub kolekcja)
– Value – może być albo wartością atomową albo
referencją do kolekcji węzłów OEM
• Jest zwykle reprezentowany jako:
<Object-ID Label Type Value>
38
Object Exchange Model –
przykład
<&oid1 Notowanie Set {&oid11 &oid12}>
<&oid11 NrNotowania String „4004”>
<&oid12 Rezultaty Set {&oid121 &oid122}>
<&oid121 Miejsce1 Set {&oid1211}>
<&oid1211 Utwor String „metropolis”>
<&oid122 Miejsce2 Set {&oid1221}>
<&oid1221 Utwor String „money”>
<&oid2 Notowanie Set {&oid21 &oid22}>
<&oid21 NrNotowania String „4005”>
<&oid22 Rezultaty Set {&oid221 &oid222}>
<&oid221 Miejsce1 Set {&oid2211}>
<&oid2211 Utwor String „learning to fly”>
39
Object Exchange Model – cechy
• Reprezentowany jako graf z obiektami
na wierzchołkach i etykietami na
krawędziach.
• Wszystkie wystąpienia są obiektami.
• Każdy obiekt ma swój unikalny
identyfikator (oid).
• Rozróżniane są dwa typy obiektów:
atomowe i złożone.
40
Object Exchange Model – cechy
(cd)
• W OEM występują tzw. nazwy (ang.
names), które mogą być traktowane jako
aliasy do obiektów wewnątrz bazy
danych.
• Nazwa służy jako wskaźnik do bazy
danych.
• Każdy obiekt w bazie danych powinien
być osiągalny za pomocą nazwy.
41
ACeDB
• ACeDB (A C. elegans Database) była
rozwijana jako baza danych informacji
genetycznej organizmów.
• Rozwijana od 1989.
• Posiada swój własny język zapytań AQL Acedb Query Language
• http://www.acedb.org/
42
ACeDB – cechy
• Schemat i dane mogą być traktowane jako
drzewo z etykietowanymi krawędziami.
• Krawędzie mogą być etykietowane
jakimkolwiek typem podstawowym. (int,
Notowanie)
np.: array Int unique Int
• Z określonego wierzchołka drzewa danych
może wychodzić wiele gałęzi.
• ACeDB pozwala na to aby jakakolwiek etykieta
różna od etykiety głównej była pominięta.
• Identyfikatory obiektów wprowadzane są przez
użytkownika.
43
ACeDB – cechy (cd)
• ACeDB wymaga schematu
• Mimo to, fakt, że dane mogą być
pomijane oraz to, że etykietowane dane
są traktowane jednolicie z innymi
prostymi typami powoduje, że jest on
bardzo bliski semistrukturalnemu
modelowi danych.
44
ACeDB – przykład
>Book
title
UNIQUE
authors
chapters
language
date
Text
Text
int
UNIQUE Text
UNIQUE english
french
other
UNIQUE month
Int
year
Int
&hock2 title
authors
chapters
language
”Computer Simulation Using Particles”
”Hockney”
”Eastwood”
1
”Computer Experiments”
2
”A One-Dimensional Model”
...
english
45
XML
• Znany i lubiany
46
Różnice pomiędzy XML a OEM
• XML jest uporządkowany.
• Etykiety w OEM są wykorzystywane tylko jako
punkt odniesienia oraz do oznaczania
zależności pomiędzy obiektami. W XML każdy
element nie będący ciągiem tekstowym zawiera
identyfikujący go znacznik – etykietę.
• XML nie wspiera bezpośrednio struktury
grafu.
47
Overview
• Semistructured Data Representation
• Mediator Generation
• Example of mediator specification
• Language expressiveness
• Implementation and performance
• Wrapper Generation
• Capabilities-Based Rewriting
48
Merge Information Relating to a
Faculty
• Schema
Integration
• Info fusion
faculty
name “John Doe”
rank “professor”
papers
...
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
49
Mediator Specification Example
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
faculty
name “John Doe”
rank “professor”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
50
Mediator Specification Example:
Semantics of Rule Bodies
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
faculty
name “John Doe”
rank “professor”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
51
Mediator Specification Example:
Semantics of Rule Heads
“John Doe”
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
faculty
name “John Doe”
rank “professor”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
52
Incrementally Add to Semantically
Identified Object
“John Doe”
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
faculty
name “John Doe”
rank “professor”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
53
Irregularities & Incomplete
Schema Knowledge
“John Doe”
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
“Mary Smith”
faculty
name “Mary Smith”
project “Air DB”
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
s1
faculty
name “John Doe”
rank “professor”
papers
faculty
name “Mary Smith”
project “Air DB”
s2
person
name “John Doe”
birthday “April 1”
54
Second Rule Attaches More
Subobjects to View Objects
“John Doe”
faculty
name “John Doe”
rank “professor”
birthday “April 1”
papers
...
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
faculty
name “John Doe”
rank “professor”
papers
...
s1
s2
person
name “John Doe”
birthday “April 1”
55
The OEM object structure of the
cs wrapper
<&e1, employee, set, {&f1,&l1,&t1,&rep1}>
<&f1, first name, string, 'Joe'>
<&l1, last name, string, 'Chung'>
<&t1, title, string, 'professor'>
<&rep1, reports to, string, 'John Hennessy'>
<&e2, employee, set, {&f2,&l2,&t2}>
<&f2, first name, string, 'John'>
<&l2, last name, string, 'Hennessy'>
<&t2, title, string, 'chairman'>
. . .
<&s3, student, set, {&f3,&l3,&y3}>
<&f3, first name, string, 'Pierre'>
<&l3, last name, string, 'Huyn'>
<&y3, year, integer, 3>
. . .
56
The OEM object structure of whois
<&p1, person, set, {&n1,&d1,&rel1,&elm1}>
<&n1, name, string, 'Joe Chung'>
<&d1, dept, string, 'CS'>
<&rel1, relation, string, 'employee'>
<&elm1, e_mail, string, 'chung@cs'>
<&p2, person, set, {&n2,&d2,&rel2}>
<&n2, name, string, 'Nick Naive'>
<&d2, dept, string, 'CS'>
<&rel2, relation, string, 'student'>
<&y2, year, integer, 3>
...
57
Object exported by med
<&cp1, cs_person, {&mn1,&mrel1,&t1,&rep1,&elm1}>
<&mn1, name, string, 'Joe Chung'>
<&mrel1, rel, string, 'employee'>
<&t1, title, string, 'professor'>
<&rep1, reports_to, string, 'John Hennessy'>
<&elm1, e_mail, string, 'chung@cs'>
58
Problemy występujące przy tworzeniu
Specyfikacji Mediatora
•
•
•
•
Schema-domain mismatch
Schematic discrepancy
Schema Evolution
Structure Irregularities
59
(MSL) Rules:
<cs_person {<name N> <rel R> Rest1 Rest2}>
: <person {<name N> <dept 'CS'>
<relation R> | Rest1}>@whois
AND decomp(N, LN, FN)
AND <R {<first name FN>
<last name LN> | Rest2}>@cs
External:
decomp(string,string,string)(bound,free,free)
impl by name_to_lnfn
decomp(string,string,string)(free,bound,bound)
impl by lnfn_to_name
60
(MSL) Rules:
<cs_person {<name N> <rel R>}>
: <person {<name N> <dept 'CS'>
<relation R>}>@whois
<cs_person {<name N> <rel R>}>
:- decomp(N, LN, FN)
AND <R {<first name FN>
<last name LN>}>@cs
<cs_person {<name N> <rel R> <title T>}>
:- ???
61
(MSL) Rules:
<cs_person {<name N> <rel R> <e_mail E>}>
:- ???
??? :- ??? <R {<first_name FN>
<last_name LN> <title E>
<reports_to S>}>@cs
Rewriting:
<cs_person {<name N> <rel R> <e_mail E> <title E>
<reports_to S>}>
:- ???
62
Language Expressiveness
• Information fusion problems solved by MSL
–
–
–
–
–
Irregularities
Incomplete knowledge of source structure
Transformation of cross-referenced structures
Inconsistent and redundant data
Use of arbitrary matching criteria
• Theoretical analysis of expressiveness
– Consider the relational representation of OEM
graphs. Then MSL is equivalent to “SQL +
special form of transitive closure”
63
Inconsistent and Redundant
Information
“John Doe”
faculty
name “John Doe”
rank “associate”
rank “assistant”
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
AND NOT <faculty {<name N> <L V1>}>@s1
s1
faculty
name “John Doe”
rank “associate”
s2
person
name “John Doe”
rank “assistant”
64
Overview
• Semistructured Data Representation
• Mediator Generation
• Example of mediator specification
• Language expressiveness
• Implementation and performance
• Wrapper Generation
• Capabilities-Based Rewriting
65
Mediator Specification Interpreter
Architecture
Result
Query
Query Rewriter
Mediator
Specification
logical datamerge
program
Cost-Based Optimizer
plan
Datamerge Engine
Queries to
Wrappers
Results
66
Query Rewriting When Known
Origins of Information
• <N faculty {<salary S>}> ::- <faculty {<name N> <salary S>}>@s1
• <N faculty {< rank R >}>
:- <person {<name N> <rank R>}>@s2
• <well-paid {<name N> <salary X>}>
:- <N faculty {<salary X> <rank assistant>}>
AND X>65000
67
Query Rewriter Pushes Conditions
to Sources
• <N faculty {<salary S>}> ::- <faculty {<name N> <salary S>}>@s1
<N faculty {< rank R >}>
:- <person {<name N> <rank R>}>@s2
• <well-paid {<name N> <salary X>}>
:- <N faculty {<salary X> <rank assistant>}>
AND X>65000
• logical datamerge program
<well-paid {<name N> <salary X>}>
:- (<faculty {<name N> <salary X>}>
AND X>65000)@s1
AND <person {<name N> <rank assistant>}>@s2
68
Passing Bindings & Local Join
Plans
Passing Bindings
s1
<salary X> :<faculty {<name $N>
<salary X>}>
AND X>65000
s2
<name N> :- <person
{<rank assistant>}>
Local Join
<a {<s X> <n N>}>:<faculty {<name N>
<salary X>}>
AND X>65000
s1
s2
N
<name N> :- <person
{<rank assistant>}>
69
Query Decomposition When
Unknown Origins of Information
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
<X faculty {<S Y>}> :- <X faculty {<birthday “1/20”> <S Y>}>
70
Plan Considers All Possible
Sources of birthday
<N faculty {<L V>}> :- <faculty {<name N> <L V>}>@s1
<N faculty {<L V>}> :- <person {<name N> <L V>}>@s2
<X faculty {<S Y>}> :- <X faculty {<birthday “1/20”> <S Y>}>
s1
birthday
s2
name
name
birthday
71
Overview
•
•
•
•
Semistructured-Data Representation
Mediator Generation
Wrapper Generation
Capabilities-Based Rewriting
72
Query Translation in Wrappers
SELECT * FROM person
SELECT * FROM person
WHERE name=“Smith”
Wrapper
Query Translator
Result
Translator
find -all
find -n Smith
Source
73
Rapid Query Translation Using
Templates and Actions
SELECT * FROM person
SELECT * FROM person
WHERE name=“Smith”
Template
Interpreter
SELECT * FROM person
{emit “find -all” }
SELECT * FROM person
WHERE name=$N
{emit “find -n $N”}
Result
Translator
find -all
find -n Smith
Source
74
Description of Infinite Sets of
Supported Queries
• uses recursive nonterminals
• Example:
– job description contains word w1 and word w2
and ...
– SELECT subset(person) FROM person
WHERE \CJob
\CJob : job LIKE $W AND \CJob
\CJob : TRUE
75
Overview
•
•
•
•
Semistructured-Data Representation
Mediator Generation
Wrapper Generation
Capabilities-Based Rewriting
76
Capabilities-Based Rewriter in
Mediator Architecture
Query
Mediator
Specification
Query
Rewriter
logical datamerge program
Cost-Based supported CapabilitiesOptimizer
plans
Based
optimal plan
Rewriter
Datamerge
Engine
Wrapper
Supported
WrapperQueries
Supported
Queries
Description
Description
77
Capabilities-Based Rewriter Finds
Supported Plans
SELECT * FROM A
WHERE salary>65000
SELECT * FROM A
Supported Queries
78
Capabilities-Based Rewriter Finds
Most-Selective Supported Plans
SELECT * FROM B
WHERE salary>65000
SELECT * FROM B
WHERE salary >65000
SELECT * FROM B
Supported Queries
79
Capabilities-Based Rewriter
Architecture
Query
Query Capabilities
Description
Component SubQuery
Discovery
Component SubQueries
Plan Construction
Plans (not fully optimized)
Plan Refinement
Algebraically optimal plans
80
What TSIMMIS Achieved
• system for integration of heterogeneous
sources
• challenges and solutions
– semistructured data & incomplete schema
knowledge
• appropriate specification language and query
processing algorithms
– limited and different query capabilities
• query translation algorithm
• capabilities-based query rewriting algorithm
81
Overview
• TSIMMIS’ goals, technical challenges,
and solutions
• Insufficiencies of the TSIMMIS’
framework
• Going forward
82
Insufficiencies of the TSIMMIS
framework
• OEM was really unstructured data
– some loose and partial schematic info may
pay off tremendously
• too “databasy” user/mediator/source
interaction
83
Overview
• TSIMMIS’ goals, technical challenges,
and solutions
• Insufficiencies of the TSIMMIS’
framework
• Going forward
84
Web emerges as a Distributed DB
and XML as its Data Model
XMAS Query
Language
XML View
Document(s)
Data
Source
Also export:
1. Schemas & Metadata
(XML-Data, RDF,…)
2. Description of
XML View
supported queries
Document(s)
Native XML
Database
XML View
Document(s)
Wrapper
Legacy
Source
85
Definition of Integrated Views
Integrated
XML View
View Definition in
XMAS
Mediator
XML View
Document(s)
XML View
Document(s)
Data
Source
Data
Source
XML View
Document(s)
Data
Source
86
Non-Materialized Views in the
MIX mediator system
Application
Blended Browsing &
Querying (BBQ) GUI
XMAS query
Integrated
View DTD
View Definition in
XMAS
XML documen
DOM for Virtual XML Doc’s
MIX Mediator
DTD
Inference
Query
Processor
Source DTD
XML Source
XML Source
87
Application
XML Document
Fragments
DOM (VXD)
Blended Browsing &
Querying (BBQ) GUI
Client API
View DTD
XMAS
Query
XMAS Mediator MIX Mediator
Resolution
View Definition
Unfolded Query
DTD
Inference
Simplification
Translation to Algebra
Optimization
DTD
XMAS
Query
XML
Execution
88