Assay Quality Considerations

Christopher N. Greene, PhD Newborn Screening and Molecular Biology Branch, Division of Laboratory Sciences NCEH, CDC

Tuesday, 9th July 2013 National Center for Environmental Health U.S. Centers for Disease Control and Prevention

Molecular Testing

Multiple molecular assays have been developed for screening procedures. These assays vary on their complexity, the target analyte, as well as the platform used for analysis.

Testing Parameters

Target Analyte Platform Complexity

Examples

Genetic Mutations, Gene Dosage PCR, qPCR, bead arrays, Fluorescent Probe, DNA Sequencers Single- or multi-plex of single nucleotides, Laboratory Developed Test, ASRs, FDA-approved, Gene Sequencing, Next-Gen Sequencing

Molecular Testing

Molecular tests require verification or validation of assay results. Level of evidence required is related to the type of assay, i.e. laboratory developed test, analyte specific reagents or FDA approved assays.

When Adopting a New Test

Test Development - LDT Test Validation - ASR Test Verification – FDA approved

Required Evidence

Accuracy, precision, reproducibility, sensitivity, specificity, robustness Clinical and analytic performance and test limitations Identify samples or reference materials of known quantitative or qualitative values to verify test performance

Laboratory Regulatory and Accreditation Guidelines

 US Food and Drug Administration (FDA):  approves kits and reagents for use in clinical testing  Clinical Laboratory Improvement Amendments (CLIA):  Regulations passed by Congress1988 to establish quality standards for all laboratory testing to ensure the accuracy, reliability and timeliness of patient test results regardless of where the test was performed  College of American Pathologists (CAP):  Molecular Pathology checklist  State Specific Regulations

 Accuracy  Sensitivity  Specificity  Precision

Testing Parameters

 Limit of Detection  Range  Reproducibility  Robustness

Definitions

• Accuracy: Agreement between test result and the “true” result – Comparison of results between new method and a reference method – Results of new method on certified reference materials • Sensitivity: Ability to obtain positive results in concordance with a positive result of a reference method – can the test detect a positive result

Definitions

• Specificity: Ability to obtain a negative result for a qualitative test in concordance with reference method • Precision: Measure of random error in quantitative tests, the closeness of agreement usually expressed as standard deviation of CV

Definitions

• Limit of Detection: Lowest amount of analyte that is distinguishable from background or a negative control • Range: Range of test values expected for the test population. For quantitative, high and low range of target, for qualitative, possible detectable alleles of the assay

Definitions

• Reproducibility: Equivalent to precision for qualitative and semi-quantitative tests. Can be used to indicate with-in or between run agreement of test results • Robustness: Test precision, given small, deliberate changes in test conditions (e.g. incubation times, temperature variation)

Pre-Validation

Choose and evaluate assay methodology Determining analytic performance of an assay involves:  Reviewing professional guidelines and relevant literature  Stringent design/analysis of primers and probes  Quality and quantity of extracted nucleic acid  Appropriate platform for the test  Availability of controls or calibrators  Optimization of amplification and detection

Verifying Assay Specifications

• Demonstrate that laboratory can obtain specifications comparable to manufacturer – Accuracy – Precision – Reportable Results • Applies to unmodified FDA-approved tests • Verify appropriate test results for population • May require matrix appropriate materials during verification

Assay Validation

Establish that the assay fulfills its intended use with the appropriate sample type and population • Performance characteristics of laboratory developed tests • Reagents that are the components of a test that is created without regulatory approval – ASR • Use of FDA-approved tests that have been modified for the testing procedure

Modified FDA Test

• Alteration of any assay step or analyte for the test that may affect performance • Changes could include: – Type or concentration of specimen – Storage specifications – Extraction method – Amplification parameters or platforms – Use of test for other purposes

Assay Validation

 Required for:  New testing methodology  Assay modification – includes cross-checks for different makes/models of instrumentation  Applies to:  Modified FDA assays  Laboratory Developed methods

Controls for Each Run

Appropriate positive and negative controls should be included for each run of specimens being tested

Molecular Assay Controls

 Positive controls:  Inhibitors  Component failure  Interpretation of results  Sources:  Residual positive DBS  PT samples  QC materials through purchase or exchange  Negative controls:  Nucleic acid contamination

Common Molecular Assay Problems and Trouble Shooting

 Temperature errors  Template/Sequence  PCR inhibitors  Buffers  Bad dNTPs  Bad primers  Bad enzyme

Critical Molecular Assay Components

 Nucleic Acids: Prepare aliquots appropriate to workflow to limit freeze-thaw cycles  Primers and probes  dNTPs  Genomic DNA  4-8 °C: Up to one year  -20 °C: Up to seven years  Enzymes  Benchtop coolers recommended  Fluorescent reporters  Limit exposure to light  Amber storage tubes or wrap in shielding (foil)

Positive Controls

Ideally should represent each target allele used in each run May not be feasible when:  Highly multiplex genotypes possible  Systematic rotation of different alleles as positives  Rare alleles  Heterozygous or compound heterozygous specimens

Positive Controls

 Assays based on presence or absence of product  Internal positive amplification controls to distinguish true negative from false due to failure of DNA extraction or PCR amplification  PCR amplification product of varying length  Specimens representing short and long amplification products to control for differential amplification  Quantitative PCR  Controls should represent more than one concentration  Control copy levels should be set to analytic cut-offs

Quantitative PCR Controls

• Contamination controls • Calibration control with reliable low limit of quantification • Measures for consistency, especially for DBS • Measure of DNA quality – internal control

False Negative: ADO

Allele drop-out (ADO): the failure of a molecular test to amplify or detect one or more alleles  Potential causes:  DNA template concentration • Incomplete cell lysis • DNA degradation  Non-optimized assay conditions  Unknown polymorphisms in target sites  Reagent component failure  Major concern for screening laboratories  Confirmation of mutation inheritance in families is not an option

DNA Degradation

 Lane 1 + 7: 1kb size standard ladder  Lane 2: 100ng control genomic DNA  Lanes 3-5: Crude cell lysates

Question?

How can you control for presence of sufficient amount/quality of DNA for a PCR based test in a NBS lab?

PCR Amplification Controls

• Allele-specific amplification • Are there problems with this assay?

• What additional controls would be useful?

Allele 1 + 2 Allele 2 Allele 1 Reference Negative

PCR with Internal Controls

Tetra-primer ARMS-PCR Simultaneous amplification of:  Positive amplification control  Mutation allele  Reference allele Alternative to tetra-primer ARMS is to include an additional primer set to amplify a different control sequence

Allele Drop-out in PCR Testing

5’ C

gtgatgtacgaggttccat ggacatgat

G

cactacatgctccaaggtagtggag

5’

cctgtacta

C

gtgatgtacgaggttccat ggacatgat

G

cactacatgctccaaggtagtggag

Allele Drop-out in PCR Testing

5’

gatgtacgaggttccat ggacatgat

G

c

G

ctacatgctccaaggtagtggag SNP in primer site

C

gtgatgtacgaggttccat

5’

ggacatgat

G

c

G

ctacatgctccaaggtagtggag

False Negatives: Deletions

Forward Primer A Reverse Primer Forward Primer G Reverse Primer

False Negatives: Deletions

Forward Primer A Reverse Primer Forward Primer G Reverse Primer

Deletion

False Positives

 Potential causes:  Non-optimized assay conditions  Unknown polymorphisms in target sites  Gene duplications  Oligonucleotide mis-priming at related sequences  Psuedogenes or gene families  Oligonucleotide concentrations too high  Nucleic acid cross-contamination

How Many Samples?

It depends…

• New test or to replace an old test?

• How is the test to be used?

• What performance criteria are most important?

• How available are appropriate samples?

• For screening: daily load vs. weekly load vs. monthly load – it depends

CLSI Evaluation Protocols recognized by FDA as Consensus Standards

• EP5-A2: Evaluation of Precision Performance of Quantitative Measurement Methods • EP9-A2: Method Comparison and Bias Estimation Using Patient Samples • EP10-A2: Preliminary Evaluation of Quantitative Clinical Methods • EP12-A User Protocol for Evaluation of Qualitative Test Performance • EP15-A2: User Demonstration of Performance for Precision and Accuracy

Molecular Assay Proficiency Testing Material Sources

 CDC NSQAP  UKNEQS  EuroGentest  CAP  Maine Molecular  SeraCare  Corielle  ECACC  In-house samples  Round-robin with other NBS laboratories

Professional Guidelines

 American College of Medical Genetics (ACMG)  Standards and Guidelines for Clinical Genetics Laboratories  Clinical and Laboratory Standards Institute (CLSI)  MM01-A2: Molecular Diagnostic Methods for Genetic Diseases  MM13-A: Collection, Transport, Preparation, and Storage of Specimens for Molecular Methods  MM14-A: Proficiency Testing (External Quality Assessment) for Molecular Methods  MM17-A: Verification and Validation of Multiplex Nucleic Acid Assays  MM19-P: Establishing Molecular Testing in Clinical Laboratory Environments

Additional Sources