Transcript Firearms and Tool Mark Identification

Firearms and Tool Mark
Identification
Responding to Recent Criticisms
- Ronald Nichols, ATF FSL - SF
Joan Griffin and David
LaMagna

Daubert Challenges to Forensic
Evidence: Ballistics Next on the Firing
Line – The Champion,
September/October 2002, 20-23; 5862.
Michael Saks

Implications of the Daubert Test for
Forensic Identification Science –
Shepard’s Expert and Scientific
Evidence, 1(3), Winter 1994, 427-432.

The Coming Paradigm Shift in Forensic
Identification Science – Science,
August 5, 2005, 892-895 with Koehler
Adina Schwartz

A Challenge to the Admissibility of
Firearms and Toolmark Identifications:
Amicus Brief Prepared on Behalf of the
Defendant in United States v. Kain,
Crim. No. 03-573-1 (E.D. PA. 2004) –
The Journal of Philosophy, Science &
Law, Volume 4, December 2004.
Adina Schwartz

A Systemic Challenge to the Reliability
and Admissibility of Firearms and
Toolmark Identification – The
Columbia Science and Technology Law
Review, Volume VI, 2005.
Lisa Steele

“All We Want You to Confirm is What
You Already Know” A Daubert
Challenge to Firearms Identification –
Criminal Law Bulletin.
Schwartz Columbia STLR
Article

Addresses Concerns Raised by Others
in Comprehensive Format
Common Themes






Scientific Reliability
Lack of Representative Databases
Lack of Statistical Treatment
Lack of Adequate Proficiency Testing
No Defined Error Rates
Current Computerized Technology
Does Not Answer the Questions
Scientific Reliability



Individual Characteristics are
Comprised of Non-Unique Marks
Subclass Characteristics May be
Confused with Individual
Characteristics
Individual Marks on a Particular Tool
Change Over Time
Representative Databases


Like DNA
Issue Broached by Dr. S. Bunch,
“Consecutive Matching Striation
Criteria: A General Critique.” Journal of
Forensic Science, Vol. 45, No. 5,
September 2000, 955-962.
Statistical Treatment


Like DNA
Issue Addressed by Dr. S. Bunch,
“Consecutive Matching Striation
Criteria: A General Critique.” Journal of
Forensic Science, Vol. 45, No. 5,
September 2000, 955-962.
Proficiency Testing



Not Representative
Inadequate and Ineffective
Administration of Tests to Examiners
Examiner Bias
Error Rates


Like DNA
Claims of Infallibility and Absolute
Identifications
Computerized Technology

Does Not Address Needs of
Representative Databases and
Statistics
Scientific Foundation
Established



Discipline firmly rooted in application
of the Scientific Method
Repeated testing of hypotheses
related to identification has been
performed
Has resulted in standard statement
settings guidelines for defining
identification criteria
AFTE Theory of
Identification*




IS a standard defining identification
criteria
IS established as a theory based on
principles of the scientific method
IS a statement of the relevant
scientific community
NOT referenced by any recent
criticisms
* Theory of Identification, Range of Striae Comparison Reports and Modified Glossary Definitions
– an AFTE Criteria for Identification Committee Report. AFTE Journal, 24(2), April 1992, 336-340.
AFTE Theory of
Identification





Addresses issues of common origin
Defines observational objectives
Defines sufficiency of agreement to
establish common origin, i.e.,
identification criteria
Defines meaning of identification
Defines role of subjectivity
Common Origin

“The theory of identification as it
pertains to the comparison of tool
marks enables opinions of common
origin to be made when the unique
surface contours of two tool marks are
in ‘sufficient agreement’.”
Observational Objectives

“Specifically, the relative height or
depth, width, curvature and spatial
relationship of the individual peaks,
ridges and furrows within one set of
surface contours are defined and
compared to the corresponding
features in the second set of surface
contours.”
Identification Criteria

“Agreement is significant when it
exceeds the best agreement
demonstrated between tool marks
known to have been produced by
different tools and is consistent with
the agreement demonstrated by tool
marks known to have been produced
by the same tool.”
What an Identification
Means

“The statement that ‘sufficient
agreement’ exists between two tool
marks means that the agreement is of
a quantity and quality that the
likelihood another tool could have
made the mark is so remote as to be
considered a practical impossibility.”
Subjectivity

“Currently the interpretation of
individualization/identification is
subjective in nature, founded on
scientific principles and based on the
examiner’s training and experience.”
Scientific Reliability
Issues of Contention – Part 1:
Individual Characteristics are
Comprised of Non-Unique Marks
Individual Marks Comprised
of Non-Unique Marks

The contention – “As a result of the
overlapping individual characteristics
of toolmarks made by different tools,
examiners who assume that a certain
amount of resemblance proves that
the same tool produced both test and
evidence toolmarks may be wrong…”*
*Schwartz, Columbia STLR, p. 6.
Individual Marks Comprised
of Non-Unique Marks


The issue – Identification Criteria
Addressed by
– Theory of Identification
– Methodology
– Extensive scientific studies
– Individual training, experience and
expertise
Theory of Identification

Identification criteria has two parts
– exceeds the best agreement
demonstrated between tool marks known
to have been produced by different tools
– and is consistent with the agreement
demonstrated by tool marks known to
have been produced by the same tool
Methodology

Comparative examinations are an
efficient means by which examiners
can observe and easily document
through photomicrography similarities
and differences between patterns of
tool marks
Scientific Studies*




Controlled studies pursued according
to the tenants of the scientific method
Have included an assortment of tools
and tool manufacturing processes
Have included tools known to have
been manufactured consecutively
Have all supported appropriateness of
the Theory of Identification
*Two review articles include, Nichols, R. “Firearm and Toolmark Identification Criteria: A Review
of the Literature,” JFS 42(3), 466-474 and “Firearm and Toolmark Identification Criteria: A
Review of the Literature – Part 2.” JFS 48(2), 318-327.
Individual Training and
Experience


Vital to examiner’s ability to properly
interpret observed correspondence
Gained through hours of microscopic
comparisons directed at gaining
familiarity of correspondence to be
expected in known non-match
situations of striated and impressed
tool marks
Contention – Small Tool
Marks

Schwartz - See e.g., John E.
Murdock…(stating that a “considerable
amount of agreement” among striated
toolmarks made by different tools is
especially likely to be found “if the
width of the mark being compared is
quite small [say, two millimeters or
less]”)…*
*Schwartz, at 6-7, n. 13.
Contention – Small Tool
Marks


Murdock never said or implied this*
2mm mark came from Butcher & Pugh+
and had nothing to do with expected
high likelihood of similarity, but rather
was simply defined as the size of a mark
likely to have enough individual detail to
permit a determination of common origin
*Murdock, J.E. “Some Suggested Court Questions to Test Criteria for Identification
Qualifications.” AFTE Journal, 24(1), 69-75.
+Butcher, S. and Pugh, D. “A Study of Marks Made by Bolt Cutters.” JFSS, 15(2), 120.
Contention – High Percentage
of Matching Striations in KNM

Screwdrivers
– Schwartz – up to 25%
– Reference is Burd and Kirk* in which the
25% is from a single KNM comparison
and in which they say the number of
matching striations (irrespective of
position) as a whole is a poor indicator
because tool marks can have high line
densities that would logically result in
more coincident matching of striations
*Burd, D. and Kirk, P., “Tool Marks: Factors Involved in Their Comparison and Use as
Evidence,” Journal of Police Science, 32(6), 465.
Contention – High Percentage
of Matching Striations in KNM

Bolt Cutters
– Schwartz – 28%
– Reference is Butcher and Pugh* in
which the 28% is the highest of a
total of 880 KNM comparisons in
which only three approached the
level of 28%
*Butcher, S. and Pugh, D. “A Study of Marks Made by Bolt Cutters.” JFSS, 15(2).
Contention – High Percentage
of Matching Striations in KNM

Bullets
– Schwartz – 15-20%
– Reference is Biasotti* who said, “…even
under such ideal conditions the average
percent match for bullets from the same
gun is low and the percent match for
bullets from different guns is high, which
should illustrate the limited value of
percent matching lines without regard to
consecutiveness.”
*Biasotti, A., “A Statistical Study of the Individual Characteristics of Fired Bullets.” JFS, 4(1).
Scientific Reliability
Issues of Contention – Part 2:
Subclass Characteristics May be
Confused with Individual
Characteristics
Subclass Characteristics

Defined as “Discernible surface features of an
object which are more restrictive than CLASS
CHARACTERISTICS in that they are: (1) Produced
incidental to manufacture; (2) Are significant in that
they relate to a smaller group source (a subset of
the class to which they belong); (3) Can arise from
a source which changes over time. Examples would
include: bunter marks, extrusion marks on pipe,
etc. Caution should be exercised in distinguishing
subclass characteristics from INDIVIDUAL
CHARACTERISTICS”*
*“Theory of Identification, Range of Striae Comparison Reports and Modified Glossary Definitions
– an AFTE Criteria for Identification Committee Report.” AFTE Journal, 24(2), 336-340.
Subclass Characteristics

The contention – “A tool may also be
wrongly identified as the source of a
toolmark it did not produce if an
examiner confuses subclass
characteristics shared by more than
one tool with individual characteristics
unique to one and only one tool.”*
*Schwartz, p. 8.
Subclass Characteristics


The issue – potential for confusion
with individual characteristics
Addressed by
– Studies of manufactured tools
– Knowledge of manufacturing processes
– Knowledge of tool use and how it relates
to orientation of any potential subclass
characteristics
Newly Manufactured
Tools

The contention – “Despite their
knowledge of this variation, firearms
and toolmark examiners have not
formulated any generalizations or
statistics about which types of tools
can be expected to produce toolmarks
with subclass or individual
characteristics when they are newly
manufactured.”*
*Schwartz, p. 9.
Newly Manufactured
Tools


Nineteen different references address
the issue of subclass characteristics
These references allow examiners to
move from knowledge of
manufacturing processes to
generalizations regarding subclass
potential to specific case application
Subclass References



Observed on consecutively made,
broach-cut rifle barrels from Cooey*
Observed on consecutively made,
broach-cut rifle barrels from
Winchester (after absorbing Cooey)+
Observed on groove impressions of a
number of F.I.E. revolvers#
*Churchman, J. The Reproduction of Characteristics in Signatures of Cooey Rifles. R.C.M.P.
Gazette, 11(5), 133-140.
+Skolrood, R. “Comparison of Bullets Fired From Consecutively Rifled Cooey .22 Calibre
Barrels.” Canadian Society of Forensic Science Journal, 8(2), 1975, 49-52.
#Lomoro, V. “Class Characteristics of 32 SWL, F.I.E. Titanic Revolvers.” AFTE Journal, 6(2),
18-21.
Subclass References


Consecutively button-rifled barrels did
not demonstrate subclass carry-over
from one barrel to the next*
Potential for subclass characteristics
(rare in any event) directly related to
manufacturing process+
*Murdock, J. “A General Discussion of Gun Barrel Individuality and an Empirical
Assessment of the Individuality of Consecutively Button Rifled .22 Caliber Rifle Barrels.”
AFTE Journal, 13(3), 84-111.
+Biasotti, A. “Rifling Methods – A Review and Assessment of the Individual Characteristics
Produced.” AFTE Journal, 13(3), 34-61
Subclass References


While limited subclass existed for three barrels
manufactured from one button rifled blank, they did
not preclude correct identification of bullets to the
barrel from which they were fired*
Previously examined broach-cut barrels examined
afresh – subclass transferred from only some
barrels to only some groove impressions on lead
bullets, copper jacketed bullets did not display any
subclass characteristics from the barrels+
*Matty, W. “A Comparison of Three Individual Barrels Produced From One Button Rifled
Barrel Blank.” AFTE Journal, 17(3), 64-69.
+Tulleners, F. and Hamiel, J. “Sub Class Characteristics of Sequentially Rifled .38 Special
S&W Revolver Barrels.” AFTE Journal, 31(2), 117.
Subclass References

Lathe turned firing pins produced
remarkably similar concentric circles
causing examiners to look for other
markings within the firing pin*
*Matty, W. and Johnson, T. “A Comparison of Manufacturing Marks on Smith & Wesson
Firing Pins.” AFTE Journal, 16(3), 51-56.
Subclass References

Cut, stamped, and milled breech faces
can all result in subclass causing
examiners to look at other features
within the breech face markings*
*Lardizabal, P. “Cartridge Case Study of the Heckler and Koch USP.” AFTE Journal, 27(1),
49-51; Thompson, E. “False Breech Face Id's.” AFTE Journal, 28(2), 95-96; Matty, W.
“Lorcin L9MM and L380 Pistol Breechface Tool Mark Patterns.” AFTE Journal, 31(2), 134137; Lopez, L. and Grew, S. “Consecutively Machined Ruger Bolt Faces.” AFTE Journal,
32(1), 19-24.
Subclass References

CNC (Computer Numerical Controlled)
Machining
– Consecutively manufactured Ruger bolt faces
demonstrated subclass yet significant
individual features to permit identification*
– Consecutively manufactured .22 caliber
barrels with subclass on the breech ends
causing carryover of characteristics in anvil
marks+
*Coffman, B. “Computer Numerical Control (CNC) Production Tooling and Repeatable
Characteristics on Ten Remington Model 870 Production Run Breech Bolts.” AFTE
Journal, 35(1), 49-54.
+Nies, R. “Anvil Marks of the Ruger MKII Target Pistol – An Example of Subclass
Characteristics.” AFTE Journal, 35(1), 75-78.
Subclass References

Ten Consecutively
Manufactured
Extractors*
– Persistent subclass on
two surfaces
– Despite subclass on
tools, test marks did
not show subclass
*Nichols, R. “Firearm and Tool Mark Identification: The Scientific Reliability and Validity of the
AFTE Theory of Identification Discussed Within the Framework of a Study of Ten Consecutively
Manufactured Extractors.” AFTE Journal, 36(1), 67-88.
Subclass References

Other Tools
– Screwdrivers – could exist unless ground
after stamping or die process*
– Pliers – though not present from broach
cut process, would have been irrelevant
because direction of use perpendicular to
orientation of tool marks+
*Burd, D. and Kirk, P., “Tool Marks: Factors Involved in Their Comparison and Use as Evidence,”
Journal of Police Science, 32(6), 1942.
+Cassidy, F. “Examination of Tool Marks from Sequentially Manufactured Tongue and Groove
Pliers.” JFS, 25(4), 796-809.
Subclass References

Persistence across generations –
defects in molds can result in similar
marks on many items* and a defect in
a master mold could be reproduced in
many molds which in turn…+
*Kreiser, J. “Identification of Cast Bullets and Their Molds.” AFTE Journal, 17(3), 88-90.
+Miller, J. “An Introduction to the Forensic Examination of Tool Marks.” AFTE Journal, 33(3),
233-248.
Rate of Change for
Subclass Characteristics

The contention – “Nor have they
developed statistics or generalizations
about the rate(s) at which subclass
characteristics on toolmarks produced
by various types of tools can be
expected to be replaced and/or joined
by individual characteristics.”*
*Schwartz, p. 9
Rate of Change for
Subclass Characteristics

Contention is
– True, and
– Irrelevant
Can assess potential for subclass
 Then assess potential for transference
 If potential exists in both cases, examiner
should be conservative and use other features
for purposes of identification

Distinguishing Between Subclass
and Individual Characteristics

The contention – “Firearms and
toolmark examiners have also failed to
develop any rules for distinguishing
between subclass and individual
characteristics.”*
*Schwartz, p. 9.
Distinguishing Between Subclass
and Individual Characteristics

Contention is NOT true
– Previously cited 19 studies
– Specifically, “That the occurrence of
subclass characteristics in rifled firearm
barrels is a rare event that can be easily
determined by direct inspection of the
rifling or a barrel cast…”*
*Tulleners, F. and Hamiel, J. “Sub Class Characteristics of Sequentially Rifled .38 Special S&W
Revolver Barrels.” AFTE Journal, 31(2), 121.
Training and Experience Vital
to Subclass Determinations


Schwartz contends that “examiners
can rely only on their personal
familiarity”*
NOT true
– 19 studies
– Regardless, such training and experience
is vital for an examiner to assess and
evaluate potential for subclass
*Schwartz, p. 9.
Scientific Reliability
Issues of Contention – Part 3:
Changing Surface of Tools Over
Time
Tool Surface Changes

The contention is that because the
surface of a tool changes over time,
this presents a “barrier in the way of
firearms and toolmark identification’s
goal of individualization.”*
*Schwartz, p. 11.
Tool Surface Changes

NOT detrimental
– It is through use that tools acquire more
individual characteristics vital to
individualization
– Tool surfaces do not change so rapidly so
as to preclude examiner’s ability to
compare and render conclusions of
identification
Studies Concerning Tool
Surface Changes


501 bullets from a 5.56 NATO rifle –
first and last compared and identified*
Lead build-up could affect
reproducibility such that best results
obtained with bullets fired in close
sequence+
*Hamby, J. “Identification of Projectiles.” AFTE Journal, 6(5/6), 22.
+Biasotti, A. “Bullet Bearing Surface Composition and Rifling (Bore) Conditions as Variables in the
Reproduction of Individual Characteristics on Fired Bullets.” AFTE Journal, 13(2), 94-102.
Studies Concerning Tool
Surface Changes


501 test fired bullets and cartridge
cases from Raven, 25 ACP caliber
pistol – identified first and last bullets
and cartridge cases*
Lead build-up studied in 900 test
firings – bullets fired 25-50 shots apart
still identifiable+
*Shem, R. and Striupaitis, P. “Comparison of 501 Consecutively Fired Bullets and Cartridge Cases
From a .25 Caliber Raven Pistol.” AFTE Journal, 15(3), 109-112.
+Kirby, S. “Comparison of 900 Consecutively Fired Bullets and Cartridge Cases From a .455
Caliber S&W Revolver.” AFTE Journal, 15(3), 113-126.
Studies Concerning Tool
Surface Changes


5000 test fired .45 ACP caliber bullets
and cartridge cases – identified first
and last bullets and cartridge cases*
7100 test fired 5.56mm caliber
cartridge cases – ejectors from #9 and
#7060 identified+
*Ogihara, Y., et al. “Comparison of 5000 Consecutively Fired Bullets and Cartridge Cases From a
.45 Caliber M1911A1 Pistol.” AFTE Journal, 15(3), July 1983, 127-140.
+Schecter, B., et al. “Extended Firing of a Galil Assault Rifle.” AFTE Journal, 24(1), 37-45.
Studies Concerning Tool
Surface Changes


4000 test fired steel jacketed bullets –
identified first and last bullets*
25 cuts by bolt cutters in lead –
reproducible through all 25; steel
shackle cuts showed differences
attributed to metal of shackle+
*Dolleing, B. “Comparison of 4000 Consecutively Fired, Steel-Jacketed Bullets.” Presented at the
53rd Annual AAFS Seminar, Seattle, WA, February 2001 .
+Hall, J. “Consecutive Cuts by Bolt Cutters and Their Effect on Identification.” AFTE Journal,
24(3), 260-272 .
Tool Surface Changes

Tool surfaces will change
– More individual characteristics
– Not rapid enough to de-value
identification discipline
– Not significant enough to prevent ability
to make identifications
– Worse case scenario – cannot achieve an
association
Scientific Basis for
Firearms and Tool Mark
Identification Has Been
Validated
Not Validated?

Criticisms focus on DNA analogy
– Statistics to deal with uncertainty
– A more objective way
– Establishing representative databases
Uncertainty

The contention
– Absolute individualization is not possible
– Need statistics to deal with uncertainty

The claim
– The goal of the discipline is to
individualize*
– Individualization is not possible therefore,
not validated
*Schwartz, p. 2.
Uncertainty

Criticisms fail to cite AFTE Theory of
Identification
– Never claims absoluteness
– “likelihood…is so remote as to be
considered a practical impossibility”*
*AFTE Glossary. “Theory of Identification as it Relates to Toolmarks.” AFTE Journal, 30(1),
Winter 1998, 86.
Meaning of Identity

What do we mean when we make
claims of identity?
– Absolute
– Practical
– Not really sure

How do we support our claim?
– Infallible
– Training and experience?
Defining Identity

Paul Kirk*
– Identity means practical or determinable
identity only
– Cannot ever establish absolute identity
because cannot test all objects
*Kirk, P. Crime Investigation. New York: Interscience Publishers, 1953, 16.
Defining Identity

David Stoney*
– “proving uniqueness” is a “ridiculous
notion”
– “Even without theoretical models and
statistics, we can, and do, make absolute
identifications. We can apply scientific,
critical judgment, expert and informed, to
make the subjective determination of
identity (or less absolutely, of ‘very very
*Stoney, D.rare’).”
“What Made Us Think We Could Individualize Using Statistics?” JFSS. 31(2),
April/June, 1991.
Defining Identity

Important to
– Concede not absolute
– Intelligently discuss issue of uncertainty
Class characteristics alone significantly narrow
range of possibilities
 Interpretation of observed correspondence is
subjective
 Some statistical models have been developed

Objectivity

Criticisms confused with regard to
issue of objectivity/subjectivity and
how it relates to claimed identifications
(absolute)
Objectivity

Objectivity and Subjectivity
– Observations made by trained examiners
are objective
– Interpretation of objective observations is
subjective
Objectivity and CMS



CMS is not inherently more objective
than non-CMS, pattern matching
CMS is simply a means of describing
the best known non-match
The CMS criteria has been tested
through 6,000+ known non-matches
and has not been violated, BUT…
Objectivity and CMS


Every object not tested
Therefore, even with CMS there is
some level of uncertainty with regard
to what an identification means
“… likelihood…is so remote as to be
considered a practical impossibility”
Representative Databases


Criticisms focus on discrete categories
of tools
BUT, individuality is not necessarily
dependent on the tool itself, but by
the manufacturing process of the tool
surface
Representative Databases


Many types of tools, only so many
ways in which they can be
manufactured
Examining propensity for creation of
class, subclass and individual
characteristics based on manufacturing
method
Representative Databases

DNA Analogy Inappropriate
– DNA deals with subclass characteristics
only; statistics are a means for calculating
frequency of a combination of subclass
characteristics
– DNA is genetically based, therefore,
discrete populations are needed to
account for expected differences in
frequencies
Representative Databases

DNA Analogy Inappropriate
– Firearms identification based on individual
characteristics, not subclass
– Manufacturing methods are limited and
propensity for class, subclass and
individual characteristics can be inferred
from the manufacturing method
Validation

Science of firearms and tool marks
identification has been validated –
multitude of studies have shown it is
possible to distinguish between tool
marks produced by tools, even those
consecutively manufactured
Validation


Criticisms focus on validation of the
identification criteria
Issue pales when it is understood
what is meant by identity
Role of Statistics

Defining the remoteness of the
likelihood
“How remote is that practical
impossibility?”
Role of Statistics

Schwartz claims that “Firearms and
toolmark examiners do not even
attempt to answer this question.”*
*Schwartz, p. 13.
Role of Statistics


Biasotti, 1959 in examining application
of CMS to identification criteria
discussed probabilities*
Brackett, 1970, idealized striated
marks to developed probabilistic
models for CMS+
*Biasotti, A., “A Statistical Study of the Individual Characteristics of Fired Bullets.” JFS, 4(1),
Jan. 1959.
+Brackett, J. “A Study of Idealized Striated Marks and Their Comparison Using Models.”
JFSS, 10(1), Jan. 1970, 27-56.
Role of Statistics



Blackwell and Framan, 1980 – using Brackett’s
formulae similar results to Biasotti*
Deinet, 1981 – probabilities of random occurences
of matches using striated tool marks+
Uchiyama, 1988 – using large tolerances for
striation correspondence, similar results to Biasotti#
*Blackwell, R. and Framan, E., “Automated Firearms Identification Systems AFIDS: Phase I.”
AFTE Journal, 12(4), Oct. 1980, 11-37.
+Deinet, W. Studies of Models of Striated Marks Generated by Random Processes. JFSS,
26(1), Jan. 1981, 35-50.
#Uchiyama, T., “A Criterion for Land Mark Identification.” AFTE Journal, 20(3), July 1988, 236251.
Role of Statistics



Miller and Neel, 2004 – statistical significance of
various CMS runs in 1,000 known non-match
comparisons*
Stone, 2003 – mathematical model for impressed
tool marks+
Collins, 2005 – empirical study testing Stone’s
mathematical model#
*Miller, J. and Neel, M. “Criteria for Identification of Toolmarks Part III Supporting the Conclusion.”
AFTE Journal, 36(1), 7-38.
+Stone, R. “How Unique are Impressed Toolmarks?” AFTE Journal, 35(4), 376-383.
#Collins, E. “How ‘Unique’ are Impressed Toolmarks? An Empirical Study of 20 Worn Hammer
Faces.” Presented at the CAC Semi-Annual Seminar, May 2005, Oakland, CA.
Role of Statistics

Bayesian
– Arbitrary nature of likelihood ratio scale
does not define remoteness of practical
impossibility

Probabilistic
– Adding or multiplying – determination of
randomness; is it determinable?
Role of Proficiency
Testing and Error Rates
in Firearms and Tool
Mark Identification
Proficiency Tests and
Error Rates

Proficiency Testing
– Provide a general indicator of error rate
– Statistics not wholly capable of answering
question of identification criteria
validation because of non-quantitative
aspects
– Therefore,
Proficiency Testing and
Error Rates
…proficiency tests can offer to the
court a reliable practical indicator of
how often the profession, using
accepted procedures, practices and
controls, makes a false identification.*
*Grzybowski, R.; Miller, J.; Moran, B.; Murdock, J.; Nichols, R.; Thompson, R. “Firearm/Toolmark
Identification: Passing the Reliability Test Under Federal and State Evidentiary Standards.” AFTE
Journal, 35(2), 216.
Proficiency Testing and
Error Rates

Firearms
– CTS: 1978-1997 – 0.9%
– CTS: 1998-2002 – 1.0%

Tool Marks
– CTS: 1978-1997 – 1.0%
– CTS: 1998-2002 – 1.5%
Proficiency Testing and
Error Rates


Does not mean that every firearms
and tool mark identification is right
only a certain percentage of the time
BUT,
A certain percentage of reported
results included a false identification
Validation Studies and
Error Rates


Brundage, 1992 – ten consecutively
manufactured Ruger barrels; 30
laboratories – 15 unknowns identified
to correct barrel in each instance*
Hamby, 2003 – further study on
Brundage barrels, 294 respondents
with no false identifications+
*Brundage, D. “The Identification of Consecutively Rifled Gun Barrels.” Presented at the 25th
Annual AFTE Training Seminar, Indianapolis, IN, June 1994.
+Hamby, J. “An Update on the Identification of Bullets Fired From Consecutively Rifled 9mm
Ruger Pistol Barrels.” Presented at the 34th Annual AFTE Training Seminar, Philadelphia, PA,
May 2003.
Validation Studies and
Error Rates
Thompson and Wyant, 2004 – ten
consecutively manufactured knives,
103 respondents with 10 unknowns
each; 8 false identifications for
0.77%*
 Murphy and Bunch, 2003 – 360
cartridge case comparisons from Glock
*Thompson,
E. and Wyant,
R. “KIP (Knife
Identification Project).” Presented at the 34th Annual
pistols,
0
false
identifications+
AFTE Training Seminar, Philadelphia, PA, May 2003.

+Bunch, S. and Murphy, D. “A Comprehensive Validity Study for the Forensic Examination of
Cartridge Cases.” AFTE Journal, 35(2), 201-203.
Validation Studies and
Error Rates

Miller and Neel, 2004 – 1000
applications of CMS criteria to 2dimensional tool marks; no known
non-match comparison violated
criteria*
Miller, J. and Neel, M. “Criteria for Identification of Toolmarks Part III Supporting the Conclusion.”
AFTE Journal, 36(1), 7-38.
Validation Studies and
Error Rates
“A better estimation of error rate in casework
would be most rigorously achieved by the reexamination of several thousand cases where
each case was examined by a panel of experts
to achieve consensus. In the absence of a
massive increase in funding, this is unlikely to
happen.” *
*Gutkowski, S., “Error Rates in the Identification Sciences,” The Forensic Bulletin, Summer 2005,
28.
Proficiency Testing and
Error Rates

Proficiency Testing
– Assessment of
Laboratory practice
 Quality assurance
 Quality control

– Involve tools from a variety of
manufacturing methods
Proficiency Testing and
Error Rates
Proficiency testing and validation studies
are not perfect but offer a good picture
on how often the profession makes a
mistake.
Current Computerized
Technology is
Misunderstood
Computerized Technology

The contention – “…computerization
has not eliminated the risks of
misidentifications and missed
identifications by firearms as well as
toolmark examiners.” *
*Schwartz, p. 28.
Computerized Technology


Current technology, e.g., IBIS was
never intended to address the issues
raised by critics
IBIS is an INVESTIGATIVE tool
Judiciary Understands
Critical Issues
Judiciary Understands

The charge - “No court, including the
two recent courts that have excluded
particular identification testimony, has
recognized the systemic scientific
problems with the field.” *
*Schwartz, p. 32.
Judiciary Understands

The alternative explanation - such
issues have been examined by the
court and they have concluded that
they do not have the significance
attributed to them by Schwartz.
Sexton v. State - 2002
Court Recognizes Potential for
Subclass
Sexton v. State - 2002



Foundation of discipline is sound
Application in this particular case was
not
Court approached situation correctly
Sexton v. State - 2002

Identification of magazine marks on
cartridge cases
– Poor foundation for testimony
– Scant references
– No application from general to magazines
– When asked, expert could not explain
manufacturing process for magazines
People v. Hawkins 1995
Court Recognizes Importance of
Training and Experience
People v. Hawkins - 1995

Court faced with non-CMS articulation
of identification criteria offered by the
prosecution and Biasotti’s CMS work
offered by the defense
People v. Hawkins - 1995

Court correctly
– Deemed that Biasotti’s work was not at
odds with testimony
– Recognized value of training and
experience and how Biasotti’s work fit in
that scheme
Commonwealth v. Ellis 1974
Court Properly Contends with
Differing Identification Criteria and
Rate of Change of Tool Surface
Commonwealth v. Ellis 1974

Case circumstances
– No firearm recovered
– Bullet from victim compared with bullets
from a tree connected to suspect
– Two experts
Agreed as to two bullets from tree
 Disagreed as to third

– One identification
– One inconclusive
Commonwealth v. Ellis 1974

Identification criteria
– Differing results not typical but
considering condition of evidence, not
surprising
– Interpretations of observed
correspondence is subjective
– With comparisons close to known nonmatch condition, differences are expected
Commonwealth v. Ellis 1974

Identification criteria
– Court did adequately assess the
limitations of the discipline and
appropriately assigned the task of weight
to the jury
Commonwealth v. Ellis 1974

Rate of Change of Tool Surface
– Court felt adequately addressed by
experts
– Court felt it did not present a systemic
problem
State v. Fasick – 1928
and State v. Clark -1930
Court is Capable of Assessing
Sufficiency of Science
State v. Fasick - 1928



Comparison of tool marks on branches
to a knife
Court appeared scientifically inclined
understanding that different angles
and testing procedures would result in
differences in tool marks
Court understood reasonable set of
expectations regarding proper testing
State v. Fasick - 1928


Court recognized that reasonable
expectations were not met by expert
Court correctly excluded evidence
State v. Clark - 1930

Circumstances similar to Fasick
– Same state
– Same type of tool
– Marks produced on branches

Holding of court completely different
than Fasick
State v. Clark - 1930

Court did not ignore Fasick - “The
facts in State v. Fasick…distinguish
that case from the case at the bar. In
the Fasick Case there was only one
mark on the two pictures admitted in
evidence which compared one with
the other. In the case at the bar there
are more than fifty marks appearing
on the pictures of the cut surfaces of
the fir boughs which can be identified
as appearing on the cut surfaces of
State v. Clark - 1930


Schwartz agrees with Saks that the
distinction was superficial*
Clark case was reported in literature
and differences are far from
superficial+
*Schwartz, p. 36, n. 163.
+May, L.S. “The Identification of Knives, Tools and Instruments A Positive Science.” American Journal
of Police Science, 1930, 246-259.
State v. Clark - 1930

Examiner
– Designed test cutting instrument
Reproducible
 Could vary numerous angles

– Identified runs of CMS on photographs
– Considered statistics
Differences far more than superficial.
Ramirez v. State – 1989,
1995, 2001
Court Questions Infallibility of
Firearms and Tool Mark
Identification
Ramirez v. State – 1989,
1995, 2001




Court referred to Hart’s method as
novel based on claims of infallibility
Court concerned about lack of
concrete items to consider as objective
criteria
Singly applied to this case
Evidence excluded
Ramirez v. State – 1989,
1995, 2001



Court erred in that Hart use traditional
non-quantitative comparative
technique – not novel
Infallibility, if claimed, not related to
method but rather issues of error rate
Error rate poorly articulated and nonresponsive
Ramirez v. State – 1989,
1995, 2001


Court did appear to recognize critical
issues and areas of concern
Court felt responses did not
adequately address these issues
– Does not mean they could not be
– All these issues addressed in presentation
– To disqualify evidence discipline wide is
not warranted
Conclusions
Firearms and tool mark
identification is rooted in firm
scientific foundations, critically
studied according to the precepts of
the scientific method culminating in
the AFTE Theory of Identification.
Conclusions

If accounted for, concerns of
– Differing identification criteria
– Potential for subclass characteristics
– Changes in tool surface over time
Do NOT invalidate identification
discipline as a science
Conclusions
The firearms and tool mark
identification discipline has been
validated in a manner appropriate
for evidence of the kind to be
expected in firearms and tool mark
examinations
Conclusions
Proficiency tests and error rates have
been studied and can provide the court
and community with a useful guide as to
the frequency with which
misidentifications are reported in the
community using appropriate
methodologies and controls.
Conclusions
In those instances in which the
discipline and interpretation of results
has been well-articulated, the courts
have recognized and have an
appreciation for the intricacies of
firearms and tool mark identification.
Conclusions
Critics have called into questions very critical issues
such as the potential for subclass characteristics and
identification criteria that, if not adequately addressed
by an individual examiner, could lead to an incorrect
interpretation of the observations made in a particular
case. Questioning, as they do, is valuable in probing the
sufficiency of knowledge and application on the part of
the individual examiner in a particular case. Extending
this discipline wide, however, is in definite error.