Transcript Tribal Water Quality Monitoring Data Management

Tribal Water Quality
Data Management
NCAI Exchange Network
Tribal Users Meeting
April 9, 2008
Overview







Tribal 106 Grants Requirements for Monitoring and
Assessment
Essential concepts of Water Quality Data Management
Understanding WQX
Steps to a WQX Submittal
Accessing Data
Upcoming Developments
How to decide where to begin
Tribal 106 Guidance Requirement

All tribes must submit a TAR (Tribal
Assessment Report) to continue eligibility for
106 funds (8-3)
Monitoring Strategy
 Water Quality Assessment
 STORET-compatible electronic data

106 Guidance parameters
Fundamental Programs:
- Dissolved Oxygen (DO)
- Temperature
- pH
- Turbidity
Intermediate Programs:
- Nitrogen
- Phosphorus
Mature Programs:
- Pathogens
- Macroinvertebrates
- Physical Habitat indicators
10 Elements of a Tribal Monitoring
Strategy
I.
II.
III.
IV.
V.
VI.
VII.
VIII.
IX.
X.
Monitoring Strategy
Monitoring Objectives
Monitoring Design
Core Indicators
Quality Assurance
Data Management
Data Analysis/Assessment
Reporting
Programmatic Evaluation
General Support and Infrastructure
Essentials of Data Management

Monitoring data consists of:
Where sample/measurement was collected
 When it was collected
 What was collected
 How it was collected
 Why it was collected
 Who collected it


It does not take a lot of information to create a
good data set.
How does data management fit in?
The life of a sample begins here
Field Sheets capture data
Labs generate data
Now what to do with all this data?
First, make sense of what you have
Organize data electronically
Use the tools you’re familiar with:
•Access
•Excel
•Other higher-powered
database management
system (i.e. Oracle)
Have a plan for growth
Not only will your skill grow, but
your data volume will grow. Be
prepared for it.
Find out what’s already available
Data are valuable, plan for re-use
Electronic data are more valuable than data in file cabinets
The more data are re-used, the more valuable they become
Shared data are of even higher value
•Provide for better planning decisions
•Incentivize collaborative efforts
•Make the most use of the monitoring $$s being
invested
How EPA Describes Monitoring Data
Three categories of data:



Projects (Why data were collected)
- Brief summary of monitoring plan
Monitoring Locations (Where data were
collected)
- Describe where monitoring takes place
Results (When, How and What were
collected)
- Measurements of what were monitored
WQX is designed around these concepts
ORGANIZATION
ACTIVITY GROUP
ORG Description
Activity Group Name
ORG Electronic Address
Activity Group Type
ORG Phone
Activity IDs
ORG Address
PROJECT
Project Description
Project Binary Object
MONITORING
LOCATION
Monitoring Location
Identity
Monitoring Location
Geospatial
Monitoring Location
Binary Object
MONITORING
ACTIVITY
Activity Description
Activity Location
Sample Description
RESULT
Sample Prep
Result Description
Subsample Description
Result Binary Object
Activity Binary Object
Result Analytical Method
Result Lab Info
Result Detection
Quantitation Level
Lab Sample Prep
The WQX Schema




The WQX schema is a definition of the structure and
format that data must have to be submitted through
WQX.
Primary components: Organization, Project,
Monitoring Location, Monitoring Activity, Samples, and
Results
Data elements currently accommodate physical,
chemical, and fish tissue data
Biological monitoring and habitat data flow will be
incorporated in the very near future
Submitting data using WQX


Water Quality Exchange (WQX) uses the
Exchange Network to flow physical, chemical
and fish tissue, and soon biological and habitat
data from an tribe’s own database to the existing
STORET Warehouse
Tribes that have been awarded Exchange
Network Grants will be able to submit data
through WQX
How WQX Works
Approved XML
Submission
XML Submission
WQX
Parse and
Load Software
CDX
Schema Validation
Receive Error Responses
Data Context Error Report
Data Load Confirmation and Summary
Data Submitter
WQX
ODS
WQX in the big picture
Steps to Submitting to WQX
1.
2.
3.
4.
5.
6.
7.
8.
9.
Review the resources that are available
Getting involved with the Exchange Network
Getting a NAAS Account
Getting an Organization ID
Understanding XML (How does my data relate
to an XML file?)
Submitting a file
Understanding errors
Understanding the WQX structure
Interacting with WQX
Step 1. Resources Available

The STORET/WQX team has several resources available to help you:

Available at the STORET web site: www.epa.gov/storet

Visit the WQX section: http://www.epa.gov/storet/wqx.html

Diagrams and presentations about how WQX works

Presentations from others on how they’ve implemented WQX
(including tribes)

Lists of allowable values for WQX

Available at the Exchange Network Site:
http://www.exchangenetwork.net/exchanges/water/wqx.htm

Read the Flow Configuration Document
See example XML Files

Examine the Data Exchange Template
Last but not least, don’t be afraid to ask for help:

Join the STORET list serve: www.epa.gov/storet/listserv.html

Call the STORET hot line: 1-800-424-9067

end an email to [email protected]


Step 1. Remember this
All of this information is available at:
www.epa.gov/storet
Step 2. Getting involved in the
Exchange Network

Ask yourself the following questions:
1.
2.
3.
4.

Does my organization have a node?
Have we submitted data for another flow?
Did we use a node? Node client? Or CDX Web?
Who is my Node Administrator? Do we have
one?
If the answer to these questions is ‘No’ or ‘I
don’t know’ visit: www.exchangenetwork.net
Step 3. Getting a NAAS Account


The NAAS or Network Authentication
Authorization Service is a user account that
you can use to access the Exchange Network.
They can be specific for an individual node or
for a particular user
NAAS accounts are assigned by Node
Administrators. If you don’t have a Node
Administrator, contact the CDX help desk at:
888-890-1995 Or [email protected]
Step 4. Getting an Org ID




An Organization ID (Org ID) is a unique
identifier assigned to your organization (i.e.
YUROK for the Yurok Tribe).
Org IDs are assigned by EPA upon the request
of the user.
You must have an Org ID to be able to submit
to WQX
To request an Org ID, send an email to
[email protected]

Provide the NAAS accounts that you want to have
access to your Organization
Step 5. Understanding XML
XML can be intimidating at first

There are many ways to create XML, some things to do:
1.
The sample files and Data Exchange Template are key to
understanding how to create a WQX XML file.
2.
Find a programmer who can spend 20-40 hours on
designing an XML file
3.
Have some test data that are representative of the data
you’ll be submitting
4.
Test the procedures that the programmer developed (plan
for an additional 20-40 hrs testing)
5.
Start small, it takes time to create an XML file, don’t waste
development time waiting for a computer to process a file.
***You should only have to do this once, after that, you’re just
running the procedure.

Step 6. Submitting a File






Once you have a NAAS account and an Org ID, and
have successfully created an XML file, you’re ready to
submit.
The original test file should be small, but
representative of the data you’ll be submitting
Submit to CDX test first
Once you get a successful submission, keep track of
the transaction ID
Send an email to the CDX help desk:…. Requesting
access to WQX production. Provide them the
transaction ID of the successful submission.
Go ahead an begin submitting to production
Step 7. Understanding Errors

There are two types of errors in WQX:
1. Schema Errors: the submitted file does not
conform to the WQX Schema. (Report is available
in the file ‘Validation Results’ which is an xml file).
2. Processing Errors: the submitted file does not
pass all of the WQX Business Rules (Report is
available in the file ‘ProcessingReport.zip’)
Step 8. Understanding the WQX
Structure

WQX files should be organized as follows:

Header (with processing instructions)

Organization





Organizational Information
Projects
Monitoring Locations
Activities
 Linkage to Projects
 Linkage to Monitoring Locations
 Results
Activity Groups
Step 9. Interacting with WQX



WQX is designed to function in a Node-toNode environment (i.e. you manage your data,
let your computer and Node keep EPA in
sync).
WQX allows Inserts, Updates, and Deletes via
a batch process
WQX has Outbound services via the Query
and Solicit method data visualization and
system synchronization purposes.
Accessing Data
The National STORET Data Warehouse serves as the
national repository for Modern STORET data as well as
WQX data. Users may obtain data from the warehouse
through the STORET home page : www.epa.gov/storet
Web services providing direct access to warehouse data
will also provide enhanced accessibility to the data.
STORET Warehouse Updates

The National STORET Data Warehouse
enhancements for improved data accessibility
New Watershed Summary – Available Now
 New Web Services – Available Now


Please visit the STORET website for more
information : www.epa.gov/storet
Where do Web Services fit?
Data in/out services via WQX
WH services available via Exch. Net.
Data out services through direct access to the WH
All Done with Web Services!!
An example using Monitoring Data
What’s coming next?

WQX 2.0 is near completion
Adds Biological data and Habitat data to the schema
 Supports Bio/Habitat metrics and indexes
 EPA will continue to support WQX 1.0 for the
foreseeable future


XML Generation Tool


Tool that will support organizations that don’t have
the ability to create XML
Batch Geo-location tool

Tie Monitoring Locations to the National
Hydrography Dataset
Dates to Remember

WQX Version 1.0 data flow in production now


Add Biological/Habitat data to WQX schema


Bio/Habitat WQX in production (May 08)
Develop Web tool for XML Generation


Physical, chemical, microbiological, and fish tissue data only
Tool available for general use (September 08)
Batch Geo-location Tool

Under development available January 2009
What is the XML Generation Tool?

A web-based application that assists a data
provider with the creation of a valid submission
file for the Water Quality Exchange.

The next generation of SIM and WebSIM,
redesigned to target WQX.
Data-sharing for all

This tool is meant to allow those who do not have Exchange
Network Nodes the ability to share data over the web

The tool will be easy to use and accept basic input formats (i.e.
Excel Spreadsheets).

Data submitted via this tool will be available on the internet
within one week of submission.
How the XML Generation Tool
fits in
CDX Web
• Online
Registration
• Log In
XML Generation Tool
• Import Flat File
• Apply Defaults & Translations
• Validate & Correct
• Export XML
State/Tribal System
State/Tribal Node
• Retrieve Data from
Database
• Export XML
• Submit
• Get Status
• Download
State/Tribal
Database
CDX Lite
• Submit
• Get Status
• Download
CDX
Node
WQX System
• Load XML File
• Update WQX Database
• Create Processing Report
WQX
Database
Demo
By the Way, what happened to National WebSIM?
Batch Geo-Locating




New development that EPA is working on
Allow for any new stations submitted via WQX to be
automatically georeferenced to the NHD+
The Georeferenced reach codes would be returned to
the data provider, as well an opportunity to QA the
georeferenced locations
This will use a concept called ‘Rain Drop’ to ensure
that points are tied to the correct NHD reach
• If the two points represent
Accuracy/Precison issues then
note how raindrop results in same
indexed location
•Note how upper point with
straight snap would go to wrong
catchment
•Note how the path does trace the
topo delineation for water
Why this is Important?

Allows for monitoring data to be easily linked to
other types of data (i.e. Impaired waters,
assessments, permits, etc.)

Allows for upstream/downstream analysis

Greatly expands the usability of monitoring data
Batch Indexing Timeline



This capability should be available January 2009
Deployed as a service that any data flow could
use (not just for WQX)
Also provide some Geospatial QA services (i.e.
is the lat/long in the county/HUC/state that the
user specified)
Where to Get Started
1.
2.
3.
4.
Figure out what data you have, and what
format it is in.
Find out where you stand with the exchange
network
Visit: www.epa.gov/storet
Give us a call, we can walk you through any
questions.
Web Resources



www.epa.gov/storet
www.exchangenetwork.net/exchanges/water/
wqx.htm
www.epa.gov/storet/wqx.html
Question/Comments

Contact:

Dwane Young, EPA (202)566-0616
[email protected]

Kristen Gunthardt, EPA (202)566-1194
[email protected]

STORET Help Desk 800-424-9067
[email protected]
Join the STORET list serve for more information,
including monthly conference calls about STORET and
WQX: www.epa.gov/storet/listserv.html