PPT Accuracy Assessment
Download
Report
Transcript PPT Accuracy Assessment
Accuracy
Assessment
Chapter 14
Significance
Accuracy of information is surprisingly
difficult to address
We may define accuracy, in a working
sense, as the degree (often as a
percentage) of correspondence between
observation and reality.
We
usually judge accuracy against existing
maps, large scale aerial photos, or field
checks.
Significance
We can pose two fundamental questions
about accuracy:
Is
each category in a classification really
present at the points specified on a map?
Are the boundaries separating categories
valid as located?
Various types of errors diminish the
accuracy of feature identification and
category distribution.
We
make most of the errors either in
measuring or in sampling.
Three types of errors dominate
Data Acquisition Errors:
These include sensor performance,
stability of the platform, and conditions of
viewing.
Reduce
or compensate for them by making
systematic corrections (e.g., by calibrating
detector response with on-board light sources
generating known radiances).
Make corrections using ancillary data such as
known atmospheric conditions, during the
initial processing of the raw data.
Data Processing Errors:
An example is misregistration of
equivalent pixels in the different bands of
the Landsat Thematic Mapper.
Geometric
correction should keep the
mismatch to a displacement to one pixel.
Under ideal conditions, and with as many as 25
ground control points (GCP) spread around a
scene, we can realize this goal.
Misregistrations
of several pixels significantly
compromise accuracy.
Scene-dependent Errors:
One such error relates to how we define
and establish the class, which, in turn, is
sensitive to the resolution of the observing
system and the reference map or photo.
Mixed pixels fall into this category.
Ancillary data
An often overlooked point about maps as
reference standards is their intrinsic or
absolute accuracy.
Maps
require an independent frame of
reference to establish their own validity.
For centuries, most maps were constructed
without regard to assessment of their inherent
accuracy.
Ancillary data
In recent years, some maps come with a
statement of confidence level.
The U.S. Geological Survey has reported results
of accuracy assessments of the 1:250,000 and
1:1,000,000 land use maps of Level 1
classifications, based on aerial photos, that
meets the 85% accuracy criterion at the 95%
confidence level.
Landsat ETM+ image
Rand McNally map
Obtainable Accuracy
Level of accuracy obtainable depends on
diverse factors, such as
the
suitability of training sites,
the size, shape, distribution, and frequency of
occurrence of individual areas assigned to
each class
which together determine the degree to which
pixels are mixed,
Obtainable Accuracy
Sensor
performance and resolution
The methods involved in classifying (visual
photointerpreting versus computer-aided
statistical classifying).
A quantitative measure of the mutual role
of improved spatial resolution and size of
target on decreasing errors appears in this
plot:
The dramatic improvement in reducing
errors ensues for resolutions of 30 m (98
ft) or better.
This
relates, in part, to the nature of the target
classes.
Coarse resolution is ineffective in
distinguishing crop types, but high resolution
(< 20 m) adds little in recognizing these other
than perhaps identifying species
As the size of crop fields increases, the error
decreases further.
The anomalous trend for forests (maximum error
at high resolution) may be the consequence of
the dictum: "Can't see the forest for the trees".
Here high resolution begins to display individual
species and breaks in the canopy that can confuse
the integrity of the class "forest".
Two opposing trends influence the behavior of these
error curves:
1) statistical variance of the spectral response values
decreases whereas
2) the proportion of mixed pixels increases with poorer
resolution.
Accuracy and Precision
Accuracy is the “correctness”
Precision is the detail
We may increase accuracy by decreasing
precision
If
we define something as “forest” it could
include pine, broadleaf, scrub, etc.
Significance
Why is it important
Legal
standing of maps and reports
Operational usefulness
Validity as basis of scientific research
Should be evaluated through a welldefined, quantitative process
This
needs to be supported by independent
evidence
Sources of error
Errors is exist in any classification
Misidentification
Excessive
generalization
Misregistration
Etc
Simplest error may be misassignment of
informational categories to spectral
categories
Sources of error
Most errors probably caused by complex
factors
Mixed
pixels
A simple landscape with large uniform
parcels is the easiest to classify
Sources of error
Important Landscape Variables
Parcel
size
Variation in parcel size
Parcel type
Number of types
Arrangement of different types
Number of parcels per type
Shapes of parcels
Radiometric and spectral contrasts
Sources of error
Errors change from region to region and
date to date
Error characteristics
Classification error – assignment of a
pixel in one category to another category
Errors
are not randomly distributed across the
image
Errors are not random to various categories
Tend to show clumped distribution on space
Errors may have spatial correlation to parcels
Occur at edges or in the interior
Map Accuracy Measurement
The task is to compare a map prepared
from RS data, with another map (reference
map) created from different source
material.
The
reference map is assumed to be accurate
If seasonal changes are important, reference
should also reflect this
Map Accuracy Measurement
Both maps must register
Both maps must use the same
classifications
Both maps must be mapped at same level
of detail
Map Accuracy Measurement
Simplest comparison is total area of each
class
Called
non-site-specific accuracy
Imperfect because underestimation in one are
can be compensated by overestimation in
another
Called inventory error
Map Accuracy Measurement
Site specific accuracy is based on
detailed assessment between the two
maps
In
most cases pixels are the unit of
comparison
Known as classification error
This is misidentification of pixels
There may also be boundary errors
Error Matrix
In the evaluation of classification errors, a
classification error matrix is typically
formed.
This
matrix is sometimes called confusion
matrix or contingency table.
In this table, classification is given as rows
and verification (ground truth) is given as
columns for each sample point.
Error Matrix
The diagonal elements in this matrix indicate numbers of
sample for which the classification results agree with the
reference data.
Off diagonal elements in each row present the numbers
of sample that has been misclassified by the classifier,
i.e., the classifier is committing a label to those samples which
actually belong to other labels. The misclassification error is
called commission error.
The off-diagonal elements in each column are those
samples being omitted by the classifier.
Therefore, the misclassification error is also called omission
error.
Error Matrix
Error Matrix
The most common error estimate is the overall
accuracy
From the example of confusion matrix, we can
obtain ω = (28 + 15 + 20)/100 = 63%.
Error Matrix
More specific measures are needed
because the overall accuracy does not
indicate how the accuracy is distributed
across the individual categories.
The
categories could, and frequently do,
exhibit drastically differing accuracies but
overall accuracy method considers these
categories as having equivalent or similar
accuracies.
Error Matrix
From the confusion matrix, it can be seen that at
least two methods can be used to determine
individual category accuracies.
(1) The ratio between the number of correctly
classified and the row total
the
user's accuracy - because users are concerned
about what percentage of the classes has been
correctly classified.
(2) The ratio between the number of correctly
classified and the column total
is
called the producer's accuracy.
Error Matrix
A more appropriate way of presenting the
individual classification accuracies.
Commission
error = 1 - user's accuracy
Omission error = 1 - producer's accuracy
Overall Accuracy
Consumer’s Accuracy
Producer’s Accuracy
Error Matrix
Error Matrix
Accuracy – e.g. Forest class
Overall . Accuracy Sum.of .diagonal 28 15 20 0.63
Grand .total
Com ission.Error
100
off .diagonal.row.elem ents 14 15
0.51
total.of .row
57
Om ission.Error
off .diagonal.colum n.elem ents 1 1
.067
total.of .colum n
30
Mapping.accuracy
diagonal. for.class
diagonal off .diag.rows off .diag.colum ns
28
0.475
28 (14 15) (1 1)
Producers accuracy = 1- 0.067=0.933
0r
93.3 %
Consumers Accuracy = 1-0.51=0.49
Or
49%
The Kappa coefficient
The Kappa coefficient (K) measures the
relationship between beyond chance
agreement and expected disagreement.
This
measure uses all elements in the matrix
and not just the diagonal ones.
The estimate of Kappa is the proportion of
agreement after chance agreement is
removed from consideration:
The Kappa coefficient
(po - pc)/(1 - pc) = (obs – exp)/(1-exp)
po = proportion of units which agree, =Spii =
overall accuracy
pc = proportion of units for expected chance
agreement =Spi+ p+i
pij = eij/NT
pi+ = row subtotal of pij for row i
p+i = column subtotal of pij for column i
K=
Error Matrix
Grand Total = 100, Total correct = 63, Observed correct = 63/100 = 0.63
Pi+ = 0.3 x 0.57 = .171,
0.3 x 0.21 = .063,
0.4 x 0.22 = 0.88
Pc = Exp correct = 0.171 + 0.063 + 0.088 = 0.322
Po = Obs correct = 0.28 + 0.15 + 0.2 = 0.63 (Overall Accuracy)
Kappa Coefficient
One of the advantages of using this method is
that we can statistically compare two
classification products.
For
example, two classification maps can be made
using different algorithms and we can use the same
reference data to verify them.
K
Two s can be derived, K 1, K2. For each K, the
variance can also be calculated.
Another Way
The following shows an alternative way to
do the error matrix
Errors
of Omission and Commission are both
calculated from the row totals in this
technique