Transcript Air Quality System (AQS): Setting Up Flows and Hands
New Tools for a Better Air Quality System Data Flow
EN 2012 May 31, 2012
Topics to Cover
• • • • Air Quality System (AQS) flow automation enhancements Exchange Network Services Center AQS Plugins for OpenNode2 and EN-Node Q&A / Discussion
Jumping Through Hoops
Can I interest you in an AQS data flow?
• • • AQS submissions via a Node or Node Client used to require extra manual steps More difficult to access AQS submission logs and reports Most users preferred to use EPA’s non-Exchange Network interface called CDX-Web
Fewer Flaming Hoops…More Puppies
Perhaps this is more to your liking?
• • • • Today, files submitted through the Exchange Network are automatically processed by AQS Exchange Network Nodes can download the submission log and reports New tools for submitting AQS data via the Network Support for AQS submissions through CDX Web is being discontinued
How Does it Work?
• • AQS data flow (v2.2a) now uses the Exchange Network Header The Header file contains information about the submission data file – Security Token – Exchange Network and AQS User IDs – Payload type: 1) XML or 2) FLAT – AQS Load Options • Final Processing Step • Stop on Error
AQS Load Options
The submitter defines the desired level of automation: 1. Stage: Submission file is uploaded to the AQS server and verified to be a valid AQS file type, but with no subsequent processing.
2. Load: The file is loaded into the AQS database and its contents validated. Sample measurements are left at the status “Pre-production”; all other data types (Site information, Monitor information, and QA information) are set to the status “Production”.
3. Post: All valid sample measurement data for the file is set to the status “Production”.
– Post is the default setting
New Submission Option #1
• Exchange Network Services Center https://enservices.epa.gov/login.aspx
– Automatically creates EN Header – Supports manual submissions to AQS – Replacing CDX-Web for AQS submissions
New Submission Option #2
Node Plugins for OpenNode2 and EN-Node • Customized for easy integration with 2 widely used air monitoring systems – OpenNode2 - .NET – AirVision and DR-DAS – OpenNode2 - Java – DR-DAS – EN-Node - .NET – AirVision – EN-Node – Java - AirVision • Automatically create EN Header • Support submissions to AQS and data publishing • In testing. Production-ready versions likely available in June for free download from EN website.
Show and Tell
• Exchange Network Services Center • EN-Node Plugin – AirVision • OpenNode2 Plugin – AirVision & DR-DAS
EN-Node AirVision Plugin
• AirVision Plugin Features 3 Operations
1
Submit AQS data to CDX
2 3
Publish AQS data (Solicit) Publish AQS data (Query)
EN Browser EPA AQS
ENDS
Solicit 2 Query 3 EN-Node AirVision Plugin CDX 1 Submit AQS Monitor AirVision
Installation File & Contents • • URL: http://code.google.com/p/en-node2/downloads/list
File: AirVisionPlugIn.zip
File Name
1. Six (6) DLL files
Purpose
Plugin software
Installation
Copy to EN-Node file directory Execute scripts 2. InsertAQSHeaderTemplate.sql
DB scripts to install AirVision Plugin in EN-Node DB 3. Task_SubmitAQSXMLData_DataWizardConfig.xml Submit AQS data to CDX 4. Query_GetAQSXMLData_DataWizardConfig.xml
5. Solicit_GetAQSXMLData_DataWizardConfig.xml
Publish AQS data (Query) Publish AQS data (Solicit) Copy to EN-Node file directory Copy to EN-Node file directory Copy to EN-Node file directory
Configure AirVision Plugin
•
What is Configuration?
– Business logics used by EN-Node to complete the data flow – Config. data saved in an external BEPL file (Business Process Execution Language) – Each Operation has its own BEPL file – Configuration tools • EN-Node Admin Console • BEPL XML file provided with the Plugin installation file •
How to Configure:
Step 1: Create a Domain (AirVision) Step 2: Register 3 operations under AirVision 1. Submit AQS data to EPA 2. Publish AQS (Query) 3. Publish AQS (Solicit) Step 3: Test Plugin Step 4: Publish AQS data at ENDS or EN Browser
Data Needed to Configure AirVision Plugin
Plugin Configuration Options Parameter Required?
Value Example EPAURL NAASUserID NAASUserPassword AirVisionURL FileName AQSXMLSchemaVersion CompressPayload DurationCode StartTime EndTime SendOnlyQAData SendRATransactions SendRBTransactions SendRDTransactions SendRPTransactions AuthorName OrganizationName DocumentTitle AQSUserID AQSScreeningGroup AQSSchemaVersion AQSPayloadTypeProperty AQSLoadOptions PayloadOperation AgencyCode SiteCode ParameterCode
Yes No No No No No No Yes No No No No No No No No No No No Yes Yes Yes No No No No Yes URL of the EPA Endpoint in which to submit data https://testcdxnode64.e
pacdxnode.net/cdx enws20/services/Netwo rkNode2 User Name Password URL of the AirVision Node File Name Version Number True/False Number in hours Date in the format YYYY-MM-DD Enfotech enfopassword http://localserver/AirVis ion.WebService
Airvision_Submission 2.2
True 1 2011-10-01 Date in the format YYYY-MM-DD True/False True/False True/False True/False True/False Author Name Organization Name Document Title AQS User Identifier AQS Screening Group Schema version of document Format in which data is presented Load/Operation Load/Operation Agency Identification Code Site Identification Code Parameter Code 2011-12-31 False True True True True John Smith Org1 Inc.
AQS Sample Enfotech Enfotech 2.0
XML Load Load 001 210 46324
Minimum Data Needed
• • • • • AirVision End Point EPA AQS End Point Organization Name NAAS ID & PWD Plugin file location
Before Configuration… • •
Important!
Open 3 DataWizardCinfig.xml files with text editor Enter Plugin file location
Step 1: Create Domain
• •
Create AirVision domain
– Choose web services (Submit, Download, Query, Solicit, Notify) allowed for the domain.
Establish
“
Administrator
”
accounts to setup/modify this domain.
Step 2: Register Operation(s)
•
Register Operation(s)
1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml
2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Define
• EPA Endpoint URL • Validation rule (if any) • Data viewing stylesheet (if any) •
Define
Task schedule
Step 2: Register Operation(s)
•
Register Operation(s)
1. Submit AQS data to EPA: Task_SubmitAQSXMLData_DataWizardConfig.xml
2. Publish AQS (Query): Query_GetAQSXMLData_DataWizardConfig.xml 3. Publish AQS (Solicit): Solicit_GetAQSXMLData_DataWizardConfig.xml
Step 3: Test Operation(s)
1. Test Submit AQS to EPA
– Test: Enter Start Date and End Date to a date earlier than the current date will execute task after the user clicks save.
– Track submission status CDX will return “ Completed ” or “ Failed ” View Download Data Viewer
Step 3: Test Operation(s)
2. Test AQS Data Query
– – Authenticate Input Start Date and End Date for AQS monitoring date – Query will return data in XML
Authentication Query Request
Step 3: Test Operation(s)
3. Test AQS Solicit
– – – – Authenticate Solicit (Input AQS monitoring date) Get Status Download AQS data (in XML)
Authentication Solicit Request Get Status Call Download
Step 4: Publish AQS Data at ENDS/ENB
• •
Manual data input to ENDS:
– Test: https://ends2.epacdxnode.net
– Production: https://ends2.epa.gov
XML Data Upload to ENDS:
– Auto Pull from ENDS: GetServices – Manual XML Upload to ENDS • • • • Test: https://ends2.epacdxnode.net/Node2WS.svc
Production: https://ends2.epa.gov/Node2WS.svc
Dataflow: ENDS_v20 Operation: Service Refresh
Data Provider
ENDS: Service/DEDL Registration
Option A: Manual A1 A2
Node Admin Registers Services (ENDS) Node Admin Registers DEDL Node Admin
EPA Service Registry (ENDS)
Exchange Network Node
Option B: Automated
(Automate your Node to transfer registration info to EPA’s Registry)
ENDS XML B1
(Published via GetServices)
DEDL XML B2
(Node submits DEDL to EPA) Stores registry of all services, input parameters, pick lists, etc.
EN Browser
Useful Tips for EN-Node AirVision Plugin
• • • •
Implement Only the Operation(s) you need Minimum data you need to configure Plugin
– AirVision End Point – EPA AQS End Point – Organization Name – NAAS ID & PWD – Plugin file location
Useful resources
– Plugin: URL: http://code.google.com/p/en-node2/downloads/list – Data Publication Guide: http://www.enbrowser.net/enbrowser/ClientBin/HowToPublishData.pdf
– ENDS & FCD: http://www.exchangenetwork.net/2011/07/15/exchange network-discovery-service-ends/
Support
– enfoTech, Lijie Xu: [email protected]
Show and Tell:
OpenNode2 AQS v2.2 Plugin
OpenNode2 AQS v2.2a Plugins
• • • http://code.google.com/p/opennode2 .NET OpenNode2 – AirVision – DrDAS Java OpenNode2 – DrDAS
OpenNode2 AQS Submit Workflow
AirVision
Or
DrDAS
Web Service 1. Request Data 2. Return Data
OpenNode2
3. Submit Data
EPA CDX Node
1. On a set schedule (or manually triggered), • request data from AirVision or DrDAS Parameters set to filter data request 2. AQS XML data is returned to OpenNode2 3. Relay the data to EPA CDX/AQS system • Optionally, header is added • Optionally, notifications can be sent
Get Status/Download Workflow
3. Notifications 1. Get Status Status Response
OpenNode2
2. Download Request Files
EPA CDX
1. On a set schedule (or manually triggered), check status of outstanding submissions 2. If CDX returns “ completed ” or “ failed ” , • download processing reports from CDX Do nothing if status is “ pending ” 3. Optionally, send notification emails to agency staff 1.
Mail sent for all submissions where status changes from “ failed ” “ pending ” to “ completed ” or
Publishing AQS Data from OpenNode2
• • “ Publishing ” makes data available for any authorized partner to come and query from your node – Exposed as Query and/or Solicit services No additional setup needed to make these services available from OpenNode2 – Setting up the services in Node Admin tool makes them available with a simple checkbox
Setting up the AQS Plugin/Flow
• • Very similar to setting up any other OpenNode2 data flow Using the Node Admin interface: 1. Create the AQS flow 2. Upload the plugin (contains all workflow logic) 3. Configure the Services (exposes plugin logic) a) b) Service 1: Get data from AirVision/DrDAS for submission to EPA Service 2: Check status of submission and download results 4. Create the Schedules (automates workflow) … and that ’ s it for the OpenNode2 portion
OpenNode2 AQS Service Setup
View in Node Admin of AQS flow with services configured: Flow 1 Service 1 Service 2 Flow 2 Service 1 Service 2
AQS
“
GetXmlData
”
Service Detail
Note: AirVision settings shown DrDAS settings very similar
AirVision Schedule Detail
DrDAS Schedule Detail
Parameter FileGenerationPurposeCode BeginDate BeginTime EndDate EndTime DataValidityCode DataApprovalIndicator IncludeMonitorDetails IncludeEventData SchemaVersion Value
“
AQS
”
Desired monitoring period start date
“
00:00
”
Desired monitoring period end date
“
23:59
” “
V
” “
Y
” “
N
” “
FALSE
” “
2.2
”
Questions?
Kurt Rakouskas 301.531.5186