Transcript MonetDB spatial

MonetDB spatial
Een open-source Spatial DBMS
Media Plaza/Het gebruik van OS software in GEO-ICT
Wilko Quak
2007-10-16
1
Onderzoeksinstituut
OTB
MonetDB
spatial
Overzicht
•
•
•
•
•
•
TUDelft / OTB / Sectie GISt / GeoInfoNed Project
De spatial DBMS markt.
Categorie-en van DBMS gebruikers
Intermezzo: spatial DBMS.
MonetDB en MonetDB (spatial)
De toekomst van MonetDB (spatial)
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
2
GIS
technology
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
3
GISt visie
Spatial Information Infrastructure (SII), a vision…
• Heading towards an information society with varying
impact on sectors of our economy
• Impact on the geo-information sector is huge: geo-info
and processing can be transported via (wireless)
networks and can be combined with positioning
• Engines of the SII nodes: geo-DBMSs (Data Base
Management Systems) filled with geo-data
• Fair pricing of data and services, no more endless
copying of data (and loss of quality/up-to-dateness)
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
4
GISt: scientific/technological goal
• Central research theme geo-DBMS
as ‘glue’ between:
•
•
•
•
•
3D spatio-temporal modeling
Computational geometry
Distributed GI processing
Mobile GIS (LBS)
Knowledge engineering
Ambition: top 1 (3) of geo-DBMS
Universities in the world
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
5
Geo Database Management Center
•
•
•
•
TU Delft open R&D center ‘geo-DBMS’
Partnerships with ICT industry: Oracle, ESRI, Bentley
‘Database lab’ and ‘3D GIS/VR lab’
Website http://www.gdmc.nl for
• Communication/
documentation
• Internet GIS
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
6
Met welk onderzoek kan de Sectie
GISt bijdragen?
• Onderzoek naar bruikbaarheid bestaande producten
(benchmarken)
• Wat mist er nog
• Meedenken met ontwikkelingen
• Eigen DBMS ontwikkelen als research vehikel:
GeoInfoNed Project
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
7
GeoInfoNed – RGI-232
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
8
wie
• CWI – Leading DBMS experts with
MonetDB
• TUDelft/OTB – Knowledge of spatial
processes
• CycloMedia – Huge dataset and
interesting problems
• RWS/AGI – Large and diverse
datasets and interesting problems
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
9
Wat
We gaan een spatial DBMS
bouwen en daar onderzoek
mee doen.
• Is er nog wel ruimte op de markt? Een inventarisatie:
• Aanbieders
• Gebruikers
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
10
De spatial DBMS markt (aanbieders)
•
•
•
•
•
Oracle – Marktleider commerciele segment
Informix/DB2 – Prima implemenatie. Niet erg actief
PostgreSQL/PostGIS – ‘Marktleider’ open-source
SQLServer – komt binnenkort
MySQL – open-source, beperkte functionaliteit
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
11
Spatial DBMS markt (gebuikers)
4 classes:
1. Server Builders: publish spatial data via web server
2. GIS User: Load various datasets and perform
complex analyses
3. Data Maintainer: Maintain one core dataset
4. Power Users: All of the above and more
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
12
Class 1: Web Server Builders
• Runs under GeoServer
• You do not really need a DBMS for this (You use a
fraction of DBMS functionality)
• Only one query counts: Find everything within BBOX.
• Multi-User
• Small Queries
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
13
Class 2: GIS users
• Full fledges GIS is much more powerful but SQL
experts might be tempted
• Main interest is functionality
• Spend more time on loading data
• Need a good query optimizer
• Data mining
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
14
Class 3: Dataset Maintainers
•
•
•
•
Limited number of queries.
24/7 stability
Transactions.
Clustering of data after updates is interesting.
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
15
Class 4: Power users
• All of the above and more
• Take time to understand software
• Run into research problems:
• Point clouds from laser-scanning
• Support 3d-volumes
• Realtime dynamic data
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
16
Intermezzo: Spatial DBMS
• Spatial data types
• Operaties op types
• Indexeren op types
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
17
Spatial Data Types
• Basic types: Point Line Polygon
• More Complex: 3D, Geodetic Coordinates
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
18
Spatial Data types volgens OpenGIS
Point
LineString
LinearRing
Polygon
MultiPoint
MultiLineString
MultiPolygon
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
19
Functions op spatial types
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
20
Indexeren spatial types
• R-tree puts MBR on object
- MBR : minimum bounding rectangle
- Object can be of type point, line or region
- At lowest (leaf) level objects are grouped to
larger MBR’s, which can be grouped (and so
on .....)
D
E
F
G
L
C
N
2007-10-16
Onderzoeksinstituut OTB
H
A
M
I
K
A B C
J
D E
B
F G
H I
J
K
L M N
Branching factor 4
Einde Intermezzo
MonetDB spatial
21
MonetDB Introduction*
MonetDB is opens
• Hardware trends
• MonetDB design considerations
• MonetDB architecture
*Slides borrowed from CWI
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
22
Hardware Trends
50% p/year:
- cpu speed
- mem size
- mem
bandwidth
- disk
bandwidth
1% p/year:
- mem latency
10% p/year:
- disk latency
Onderzoeksinstituut OTB
Latency is the enemy!
• Commercial DBMS products (oracle, DB2, SQLserver)
stem from OLTP roots
• focus on minimizing random I/Os => depend on
latency!
• MonetDB: built for bulk access
• optimize CPU and memory performance
Onderzoeksinstituut OTB
MonetDB design considerations
•
•
•
•
•
•
•
•
Multi-model database kernel support
Extensible data types, operators, accelerators
Database hot-set is memory resident
Simple data structures are better
Index management should be automatic
Do not replicate the operating system
Optimize when you know the situation
Cooperative transaction management
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
25
MonetDB product family
End-user application
SQL
JDBC
ODBC
XQuery
Python
Perl
PHP
C-mapi lib
MAPI protocol
2007-10-16
Onderzoeksinstituut OTB
Monet
kernels
MonetDB spatial
26
SWOT
strength
weakness
State-of-the-art SQL & XQuery
DBMS; multi-model database
support; decomposed storage model;
query processing for decision support
systems; extensible kernel; multiple
APIs;…..
Triggers and SQL procs missing;
XQuery; some overhead in simple
SQL queries; MIL scripting; missing
GIS module
threats
opportunities
Commercial systems requirements;
competition established market;
floating interest; development skills
and quality awareness
Unique selling points for advanced
applications in multimedia,
streaming, covering both small and
the large world
GET INSPIRED BY ADVANCED APPLICATIONS
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
27
MonetDB - Physical data organization
• Binary Association Tables
ID
10
11
12
13
OID
ID
100
101
102
103
104
2007-10-16
Onderzoeksinstituut OTB
OID
10
11
12
13
14
Day
Discount
4/4/98
0.195
9/4/98
0.065
1/2/98
0.175
7/2/98
0
Day
100
4/4/98
101
9/4/98
102
1/2/98
103
7/2/98
104
1/2/99
OID
MonetDB spatial
Discount
100
0.195
101
0.065
102
0.175
103
0
104
0.065
28
Spatial MonetDB
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
29
Spatial MonetDB -- nu
• Eerste implementatie in volgende release (Q4).
• OpenGIS implementatie gebaseerd op PostGIS
• OpenJUMP plugin
• Shape2MonetDB converter
• GeoTools Plugin (in ontwikkeling)
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
30
Spatial MonetDB – de toekomst
• MonetDB moet het hebben van zijn speciale
architectuur.
• Uitdagingen:
• Puntenwolken van Laserscanner.
• Data-minen op grote spatio-temporele datasets.
• XML-Data.
• Saaier werk:
• Testen huidige implementatie
• debuggen
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
31
Vragen
Discussie
2007-10-16
Onderzoeksinstituut OTB
MonetDB spatial
32