Transcript PowerPoint - CUAHSI-HIS

Using Python to implement
a Hydrologic Information
System for Texas
James Seppi
(CRWR, University of Texas at Austin)
Dharhas Pothina
(Texas Water Development Board)
Ernest To
(Espey Consultants, Inc)
Tim Whiteaker
(CRWR, University of Texas at Austin)
Andrew Wilson
(Texas Water Development Board)
Texas Water Development Board



Provides leadership and scientific expertise for
development and conservation of water
Accumulate and publish large-scale State water
databases in a form accessible as WaterOneFlow web
services.
Create a Texas-centric HIS Water Data for Texas:
 Partnerships with Federal and Texas agencies to
obtain data
 Inherits National HIS technology
 Adds Texas-specific services to
provide ease of use and a level of
quality control
Why a Texas-specific HIS?

National HIS technology available but uptake low among
Texas agencies




lack of motivation
lack of time/resources
lack of experience with the technology
National HIS doesn’t quite do everything TWDB needs.



Parameter mapping
Unit Conversions
Web Viewer
original paradigm
Put your data in an
ODM database
i.e. learn swahili
this didn’t get much traction…




Data is usually already in the format the
organization needs.
Changing formats/delivery mechanisms is a
pain.
ODM DB Schema doesn’t always fit the needs of
data collecting agencies.
Resources. Resources. Resources.
Overall response: yawn
new paradigm… make it easy





Form partnerships to obtain data
Provide technical know-how and support
Provide multiple paths to serve data
Provide a useful product with branding that is
available to the general public can use.
Scheduled updates to datasets
i.e. we have trained swahili translators
available please speak to us
Four Components of
2. Translation
services to
convert nonWaterML data
sources into
WaterML
3. WDFT Central
Registry service
 Limited
Parameter Set
 List of Texas
HIS data
sources
 Unit
conversions
1. Existing WaterML
compliant web
services within
Texas
WaterML
compliant
web
services
4. Access Tools for
data discovery
and download:
HydroExcel,
HydroDesktop,
HydroGET…
GEMSS
(TNRIS hosted map based
web interface)
Why python?




Built-in cross-platform compatibility
Availability of a massive variety of scientific
libraries/modules
Amazing community support. Better than
commercial vendors
$FREE!$
Python @TWDB






pINT - Python INstrument Toolkit
SWIS - Surface Water Information System
WOFpy - Python WaterOneFlow Wrapper
pyHIS - Python HIS Client
pyHAT - Python Hydrographic Analysis Toolkit
pyselfe/pug – hydrodynamic model analysis &
visualization
WOFpy concept
WOF py
WOFpy – a python wrapper for WaterOneFlow Services
Old paradigm (i.e. ODM db as intermediate)
Data in ODM
Native
database
ODDataloader
SSIS scripts
SQL views
Manual loading, etc.
WaterML
Generic
ODwebservices
•Significant learning curve
•Data persists in ODM under Microsoft SQL server
i.e., not in native format and environment
•“When is the next update?”
•“What if data is collected in real-time?”
Old paradigm (i.e. ODM db as intermediate)
Metadata in ODM
WaterML
for GetSites,
GetVariables,
Native
metadata
Loading scripts
Realtime
data on
web
etc.
Generic
ODwebservices
WaterML
for GetValues
Customized webscraper web service
Old paradigm
(ODM
db asframework)
intermediate)
New
paradigm (i.e.
single
Metadata in ODM
Single framework
WaterML
for GetSites,
• Dynamically map between native database
GetVariables,
schema and WaterML
Native
etc.
• Allow data to persist in native database
metadata
• Accomodate different database environments
WaterML
(MSSQL, SQLite, etc)
• Handle both static and real-time data
• Dispatch and manage WaterOneFlow services
in both SOAP
WOF py
Generic
Loading scriptsand REST
ODwebservices
• Open-source and fully
customizable
WOFpy – a python wrapper for WaterOneFlow Services
Realtime
data on
web
WaterML
for GetValues
Customized webscraper web service
WOFpy architecture (backend – data access)
Data persists
in source’s
native db
environment
Native
metadata
database
WOF py
(WOF module)
WOFpy – a python wrapper for WaterOneFlow Services
• WOF module assembles
WaterML elements to WaterML
responses.
• Implements a reduced ODM
data model (‘Model’) for
mapping to WaterML elements.
• Model is also mapped to the
native db of the source by
source-specific DAOs.
Source-specific
DAOs
Realtime
data on
web
(Data Access
Objects)
WaterML
classes
• Model
DAOscontains
define object-relational
mapping (ORM)
common
to
both
from native schema to Model.
WaterML
ODM.for translating user
• Contains and
methods
arguments into queries to native db.
• The python module, SQLalchemy, supports
ORMs and can accommodate different db
environments (e.g. MSSQL, SQLite, etc)
WOFpy architecture (frontend - webservices)
Backend (WOF module)
WOFpy_flask
-Defines the REST
methods for
WaterOneFlow
using Flask
WOF py
WOFpy – a python wrapper for WaterOneFlow Services
WaterML
Native
metadata
database
-Creates URLs and
views for both
REST and SOAP
web services.
REST
web
service
Source-specific
DAOs
Realtime
data on
web
(Data Access
Objects)
WOFpy_soap
-Defines the SOAP
methods for
WaterOneFlow
using SOAPlib
• The python libraries provide capabilities
to serve REST and SOAP. Both types
access the same backend.
• WOFpy web services can be accessed
by CUAHSI HIS clients just like any
other WaterOneflow web service.
SOAP
web
service
Runserver_all.py
-Management script
-Launches both webservices using
Werkzeug.DispatcherMiddleware
WOFpy demo
WOF py
WOFpy – a python wrapper for WaterOneFlow Services
Next steps

Release WOFpy and other python modules as
open-source software





Developmental version on GITHub (search for
WOFpy)
Continue to add datasets to
Add parameters to Texas Registry
Conduct training workshops.
Develop how-to’s for putting data into the system
and also retrieving data. WDfT System will be live
by end of summer
WOF py
WOFpy – a python wrapper for WaterOneFlow Services




A Texas-specific HIS hosted by
the Texas Water Development
Board
Employs partnerships with
Federal and Texas agencies to
share water data
Inherits National HIS technology
Provides additional tools and
services to provide ease of use
and a level of quality control for
partners and clients
(e.g. WOFpy)




A fully customizable, opensource framework written in
Python to publish
WaterOneFlow web services
User can maintain their data in
their original database format
Mapping to waterML is defined
by user-customized Data
Access Objects (DAOs)
Tutorials and documentation
coming soon!
Acknowledgments:
James Seppi
WOFpy creator