Transcript Template for NMP print presentations

Digital Imagery Policies, Standards, and Guidelines
Inter-Agency Digital Image Working Group
October 2005
U.S. Department of the Interior
U.S. Geological Survey
Quality Assurance Plan For Acquisition of Digital
Imagery
Inter-Agency Digital Image Working Group
October 2005
U.S. Department of the Interior
U.S. Geological Survey
Bottom-Line Goal

Ensure overall quality of image products
and services to end-users
Today’s agenda




Background
The Proposed Quality Plan
Issues to be resolved
Next steps
Background




ASPRS camera calibration report
Interim solutions by USGS
Formation of Inter-agency Digital Imagery
Working Group (IADIWG)
Focus Meeting in Denver
Proposed Four Step Quality Plan




Manufacturers’ certification process
Data Providers’ certification process
Image specifications and selection of
contractors
Inspection and acceptance of deliverables
Manufacturers’ Certification Process



USGS will establish a Digital Sensor
Certification Team to review and certify
manufacturer’s sensor calibration process
Initial certification based on review of
documentation and on-site visit
Certification letter and report issued to
manufacturer
Manufacturers’ Certification Process

Core members of team will include:
USGS management official
 Expertise in radiometry, geometry, optics,
system engineering, quality, standards
 Specialized expertise, if necessary

Manufacturers’ Certification Process




Re-certifications due to calibration
process changes and engineering
modifications
Request by letter and documentation
On-site visit only when deemed necessary
Re-issued certification letter and report as
needed
Data Providers’ Certification Process



USGS will establish a Data Provider
Certification Team to review and certify
Data Provider’s sensor calibration and
quality assurance process
Initial certification based on review of
documentation and on-site visit, if
required
Re-certification required every 3 years
Data Providers’ Certification Process

Short-term:
Certify Data Provider’s calibration procedures
and initial calibration results
 Verify accuracy and image quality of specific
products (i.e., Product characterization)

Data Providers’ Certification Process

Long-term:
Development of certification guidelines and
procedures
 Use of guidelines is a requirement or factored
into the evaluation of Data Providers by the
contracting officer

Contracting Process

The basic federal contracting process is:





Seek appropriate internal authorizations and establish
availability of funding
Develop written statement of work of technical
requirements for products and services
Develop evaluation criteria for selecting contractor
The Contracting Officer (CO) determines the type of
procurement mechanism or type of contract
Publish Request for Proposal (RFP)
Contracting Process

RFPs include:





Complete statement of work of products and services
to be provided by the contract (performance based)
Contractor instructions for preparation of technical
and pricing proposals
Selection criteria and relative weighting factors
Related documents to be provided with proposal
Deadline for submittal of proposal
Contracting Process: Boiler Plate
Guidelines

Process used in developing the draft
guidelines:
All existing specifications among the IADIWG
agencies were gathered and compiled into a
single document
 Review and agreement to organization and
common terms

Contracting Process: Boiler Plate
Guidelines

Goal:
To establish a boiler plate document that could
be used by all agencies as a starting point in
image acquisition procurements
 To use common terms and common language
in specifying requirements
 To seek industry review for correctness and
understanding

Contracting Process: Boiler Plate
Guidelines

Future plans:
Incorporate changes from user community
 Provide document to other agencies via website and to State liaisons
 Establish on-line application to “write” agency
specific specification

Contracting Process

Contractor selection process:



Contract Specialists determine if proposals submitted
meet all minimum contract requirements and deadlines
Contracting Officer establishes a technical evaluation
panel with instructions to review and rate the technical
proposals
Evaluation panel provides proposal ratings and
recommendations to the CO
Contracting Process

Example of initial evaluation factors:
Project management capabilities statements
 Past performance
 Quality control system statements
 Personnel qualifications
 Aircraft and camera availability
 Workload and incomplete contracts

Contracting Process

BUT, selection factors during course of
multi-year contract tend to be:
Past performance
 Quality of past deliverables
 QA/QC of subcontractors

Contracting Process

Contract negotiation and award process:
CO clarifies any questions relating to the
proposal to ensure a fair evaluation
 CO conducts negotiations with contractors
established within the competitive range
 CO selects highest rated contractor
 CO awards the contract

Contracting Process

IADIWG recommends the following to
evolve into requirements for all contracts:
Factory calibration report for all sensors used
 Manufacturers’ certification or equivalent
 Data Providers’ certification or equivalent
 Data Provider’s latest calibration report

Contracting Process

IADIWG recommends these factors become part
of the evaluation process:



Data Provider’s sensor acceptance test plan and
results
Data Provider’s documentation of processes that
ensure sensor is properly maintained and operated
Records to indicate that procedures have been
followed and that the sensor is in good calibration
Inspection and Acceptance of
Deliverables

What is quality?
I don’t know, but I know it when I see it!
 Inspection and acceptance, and quality control
requirements are generally spelled out in the
contract specifications; however, this does not
ensure quality

Inspection and Acceptance of
Deliverables

Quality:
Does it meet contract specifications?
 Does it meet user needs?

Inspection and Acceptance of
Deliverables

What’s easy?

Objective calls:




Specific tolerances
Formatting
Media
Naming convention and labels
Inspection and Acceptance of
Deliverables

What’s hard?
Unrealistic tolerances because of improper or
impossible specifications
 Meta data
 Determining image quality (subjective calls)

Inspection and Acceptance of
Deliverables

What’s really impossible?
100% inspection
 Agreement on image quality
 Improving image quality when source is bad

Inspection and Acceptance of
Deliverables

What’s needed?





Early inspection of sample from contractor
Delivery in small lots
Practical specifications for such things as tone
matching, color balance, and edge joins
Better, less subjective, more automated means of
determining image quality (e.g., commercial NIIRS
rating)
Standardized quality assurance/quality control
methods
Issues to be resolved




Who pays and how much?
Detailed plan for certification and
recertification
ISO certification vs. IADIWG certification?
Guidelines for using local in-situ ranges
for calibration
Next steps




Invitation for comment to ASPRS and
MAPPS membership
Communicate plan to mapping community
through education, publications, and
USGS State Liaisons
Final plan presented at Reno 2006
Begin manufacturers’ certification process
For more information:

http://calval.cr.usgs.gov/
Contacts

Greg Stensaas, 605-594-2569,
[email protected]

George Lee, 650-329-4255,
[email protected]