CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008
Download
Report
Transcript CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008
CIWQS Review Phase II:
Evaluation and Final
Recommendations
March 14, 2008
Background
Review Panel convened in response to State
Water Board’s request for external review of
CIWQS
First Panel meeting held in May 2007
Preliminary Panel report with recommendations
issued July 2007
Water Board staff to consider recommendations
and demonstrate progress toward benchmarks
through February 2008
Findings of Previous Review
Seven essential recommendations:
Reduce the project’s scope
Restructure CIWQS’ project management
Validate the system’s requirements
Rebuild key constituencies
Address data quality issues
Produce key reports
Improve user interface
Goals for Phase II
Water Board staff will demonstrate progress by
responding to Phase I recommendations
Including specific actions, timelines, and additional
performance measures for the coming year
Panel will review progress and prepare final
report to:
Judge adequacy of Water Board’s response
Make final recommendations for future direction of
the project
Overview of Final Review
CIWQS remains essential for the State Water Board to
fulfill its mission
Terminating funding would be counterproductive
CIWQS should be funded at an appropriate level
The program’s new direction should lead to success
We are impressed with the remarkable turnaround since May
We are particularly impressed with progress in
reorganizing management and communicating with
users
Overview (cont.)
The redefinition of project scope was
appropriate and effective
We have remaining concerns with some
aspects of the technical approach, rate of
data cleanup, and report production
Original Recommendations
Reduce the scope
Restructure management
Validate system requirements
Rebuild constituency
Address data issues
Produce key reports
Improve user interface
Reduce the Scope
Evaluation
Achieved significant scope reduction
Increased probability of project success
Achieved buy-in from Steering Committee & users
Phased eSMR implementation plan is reasonable
Implementation through eSMR Level 3 is desirable
We concur with need for a separate federal interface
Recommendation
Consider trimming unneeded parts of the database to
meet reduced project scope
Restructure Management
Evaluation
The new management structure positions CIWQS for
success
Effective new organization with leadership in Water Quality
not in DIT ties the project closer to the Board’s mission
Buy-in from senior managers for the project’s importance
and new direction
Validate System Requirements
Evaluation
Fundamentally improved systems design & implementation
Corrected design and approach based on testing
The underlying structure can be made to work
Referential integrity issues are being addressed
Requirements better reflect user inputs & needs
Recommendations
Demonstrate a disciplined systems engineering process
Document team members’ roles, responsibilities, activities
Use test cases to demonstrate successful resolution of
referential integrity issues
Rebuild Constituency
Evaluation
Achieved remarkable cultural change that significantly
improved user communication and support
Developed effective user community structure
Users attended Panel meeting and strongly supported the
project
Users strongly engaged in most areas
Issues with stormwater group illustrate cultural
improvements have further to go
Recommendations
Should apply approach used for eSMR to all groups and
issues
Address Data Issues
Evaluation
Less was accomplished than we had expected
Acknowledged the need for systematic QA/QC
Database integrity is being addressed
Some progress on correcting existing data errors
Recommendations
Devote more focused effort to QA/QC and data cleanup
Develop coherent plan that prioritizes data cleanup and
identifies needed resources and timeframe
Create formal QA/QC mechanisms, including user
participation
Make QA/QC an integral part of database administration
and operation
Produce Key Reports
Evaluation
Created a mechanism to identify and prioritize key reports
Made some progress on developing canned reports
Recommendations
Demonstrate that required reports can be produced from
the core CIWQS system
Increase staff’s ability to create ad hoc reports
Accelerate report production
Improve User Interfaces
Evaluation
Created a mechanism for input and feedback from users
Created a successful and easy-to-use SSO module
Seem to be heading in the right direction
Recommendations
Create a formal mechanism for validating and testing user
interfaces
Create context-specific drop-down menus
Create context-specific online help
Additional Recommendations
Build a non-CIWQS interface with ICIS-NPDES
Develop a set of persistent end-to-end tests to
demonstrate that the data model and
processes work
Develop a deployment plan for the system and
data
Fund the system at an appropriate level
ICIS-NPDES Issue Must Be Resolved
ICIS-NPDES will not meet State Board’s needs
There are different reporting needs for EPA and
for State Board
State system should cross-check violation
determinations between state and federal
systems to quickly identify erroneous violations
Download ICIS-NPDES data to CIWQS as needed
for State Board purposes
Evaluate Interface Options w/EPA
Consider discharger submission using
eDMR (file-oriented)
netDMR (interactive)
Develop test plan and case(s) including EPA DMR
data download to state system
EPA has agreed to this using currently submitted data;
this should be formalized quickly to take advantage of the
offer
Develop End-to-End Testing
Develop and maintain a library of test
cases with known results that can be used
to test system and processes as changes
are made
Should be implemented as part of all
deployment plans on an ongoing basis
Develop Deployment Plan
Develop a deployment plan for every
system update and release
Should include software, hardware, data,
and quality control aspects
End-to-end testing should be integral part
of the plan
Each phase of the eSMR development
needs a deployment plan
Fund CIWQS at Appropriate Level
Panel is encouraged by progress
Believe that changes make it likely the
CIWQS project will result in a system that
serves the State Board’s needs for
accuracy, automation, and efficiency for
the future
Encourage State Board and Legislature to
fund the development at levels that can
maintain and accelerate development
Next Steps
Full report of findings and recommendations, with
performance metrics, within 2 months
Panel open to revisiting progress in 9 – 12 months if
asked
Summary
The project has made substantial improvements and
is on a path to success
Management restructuring, rebuilding user constituencies,
and system validation are clear successes
Additional progress needed on data cleanup, reporting, user
interfaces
Additional recommendations include:
Build a non-CIWQS interface with ICIS-NPDES
Develop end-to-end tests to demonstrate all functionality
Develop comprehensive deployment plan
Fund CIWQS at an appropriate level