Transcript CIWQS Review Phase II: Evaluation and Final Recommendations March 14, 2008

CIWQS Review Phase II:
Evaluation and Final
Recommendations
March 14, 2008
Background
 Review Panel convened in response to State
Water Board’s request for external review of
CIWQS
 First Panel meeting held in May 2007
 Preliminary Panel report with recommendations
issued July 2007
 Water Board staff to consider recommendations
and demonstrate progress toward benchmarks
through February 2008
Findings of Previous Review
 Seven essential recommendations:

Reduce the project’s scope

Restructure CIWQS’ project management

Validate the system’s requirements

Rebuild key constituencies

Address data quality issues

Produce key reports

Improve user interface
Goals for Phase II
 Water Board staff will demonstrate progress by
responding to Phase I recommendations

Including specific actions, timelines, and additional
performance measures for the coming year
 Panel will review progress and prepare final
report to:


Judge adequacy of Water Board’s response
Make final recommendations for future direction of
the project
Overview of Final Review
 CIWQS remains essential for the State Water Board to
fulfill its mission

Terminating funding would be counterproductive
 CIWQS should be funded at an appropriate level
 The program’s new direction should lead to success

We are impressed with the remarkable turnaround since May
 We are particularly impressed with progress in
reorganizing management and communicating with
users
Overview (cont.)
 The redefinition of project scope was
appropriate and effective
 We have remaining concerns with some
aspects of the technical approach, rate of
data cleanup, and report production
Original Recommendations
 Reduce the scope
 Restructure management
 Validate system requirements
 Rebuild constituency
 Address data issues
 Produce key reports
 Improve user interface
Reduce the Scope
 Evaluation






Achieved significant scope reduction
Increased probability of project success
Achieved buy-in from Steering Committee & users
Phased eSMR implementation plan is reasonable
Implementation through eSMR Level 3 is desirable
We concur with need for a separate federal interface
 Recommendation

Consider trimming unneeded parts of the database to
meet reduced project scope
Restructure Management
 Evaluation



The new management structure positions CIWQS for
success
Effective new organization with leadership in Water Quality
not in DIT ties the project closer to the Board’s mission
Buy-in from senior managers for the project’s importance
and new direction
Validate System Requirements
 Evaluation



Fundamentally improved systems design & implementation
Corrected design and approach based on testing
The underlying structure can be made to work


Referential integrity issues are being addressed
Requirements better reflect user inputs & needs
 Recommendations

Demonstrate a disciplined systems engineering process


Document team members’ roles, responsibilities, activities
Use test cases to demonstrate successful resolution of
referential integrity issues
Rebuild Constituency
 Evaluation





Achieved remarkable cultural change that significantly
improved user communication and support
Developed effective user community structure
Users attended Panel meeting and strongly supported the
project
Users strongly engaged in most areas
Issues with stormwater group illustrate cultural
improvements have further to go
 Recommendations

Should apply approach used for eSMR to all groups and
issues
Address Data Issues
 Evaluation




Less was accomplished than we had expected
Acknowledged the need for systematic QA/QC
Database integrity is being addressed
Some progress on correcting existing data errors
 Recommendations




Devote more focused effort to QA/QC and data cleanup
Develop coherent plan that prioritizes data cleanup and
identifies needed resources and timeframe
Create formal QA/QC mechanisms, including user
participation
Make QA/QC an integral part of database administration
and operation
Produce Key Reports
 Evaluation


Created a mechanism to identify and prioritize key reports
Made some progress on developing canned reports
 Recommendations



Demonstrate that required reports can be produced from
the core CIWQS system
Increase staff’s ability to create ad hoc reports
Accelerate report production
Improve User Interfaces
 Evaluation



Created a mechanism for input and feedback from users
Created a successful and easy-to-use SSO module
Seem to be heading in the right direction
 Recommendations



Create a formal mechanism for validating and testing user
interfaces
Create context-specific drop-down menus
Create context-specific online help
Additional Recommendations
 Build a non-CIWQS interface with ICIS-NPDES
 Develop a set of persistent end-to-end tests to
demonstrate that the data model and
processes work
 Develop a deployment plan for the system and
data
 Fund the system at an appropriate level
ICIS-NPDES Issue Must Be Resolved
 ICIS-NPDES will not meet State Board’s needs
 There are different reporting needs for EPA and
for State Board
 State system should cross-check violation
determinations between state and federal
systems to quickly identify erroneous violations
 Download ICIS-NPDES data to CIWQS as needed
for State Board purposes
Evaluate Interface Options w/EPA
 Consider discharger submission using


eDMR (file-oriented)
netDMR (interactive)
 Develop test plan and case(s) including EPA DMR
data download to state system

EPA has agreed to this using currently submitted data;
this should be formalized quickly to take advantage of the
offer
Develop End-to-End Testing
 Develop and maintain a library of test
cases with known results that can be used
to test system and processes as changes
are made
 Should be implemented as part of all
deployment plans on an ongoing basis
Develop Deployment Plan
 Develop a deployment plan for every
system update and release
 Should include software, hardware, data,
and quality control aspects
 End-to-end testing should be integral part
of the plan
 Each phase of the eSMR development
needs a deployment plan
Fund CIWQS at Appropriate Level
 Panel is encouraged by progress
 Believe that changes make it likely the
CIWQS project will result in a system that
serves the State Board’s needs for
accuracy, automation, and efficiency for
the future
 Encourage State Board and Legislature to
fund the development at levels that can
maintain and accelerate development
Next Steps
 Full report of findings and recommendations, with
performance metrics, within 2 months
 Panel open to revisiting progress in 9 – 12 months if
asked
Summary
 The project has made substantial improvements and
is on a path to success


Management restructuring, rebuilding user constituencies,
and system validation are clear successes
Additional progress needed on data cleanup, reporting, user
interfaces
 Additional recommendations include:




Build a non-CIWQS interface with ICIS-NPDES
Develop end-to-end tests to demonstrate all functionality
Develop comprehensive deployment plan
Fund CIWQS at an appropriate level