Transcript Standard Scripts - Project 2 Proposal for Qualification

Standard Scripts - Project 2
Proposal for Qualification
July 2014
Project 2 Update
Main Sections
• Summary of prior proposal, 2013
• Updated proposal, July 2014
Main Sections
Summary of prior proposal
o Concepts, definitions & meta data
o Test data considerations
o Heavy vs. Light qualification
• Updated proposal
Proposal from 2013
http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.
ppt
• Anyone should be able to submit a script, according
to a check list
• Categorize scripts according to complexity
– Complexity:
low, medium, high, software
– Output: tabulated data, analysis data, table, figure, listing
• Metadata for script should indicate
– Type of output: tabulated data, analysis data, table, figure,
listing
– Study design:
parallel, crossover, etc
– State of qualification:
?
Proposal through CSS
2104
• Test data
– Overall project should have minimum test data (SDTM & ADaM)
– Scripts can propose new test data, must pass (Data fit? Open CDISC?)
– Share program to produce test data, never binary test data
• 2 levels of qualification to match script complexity/output
– Light vs. Heavy qualification
– Common elements include
•
•
•
•
•
header
good programming practices
clearly declared scope of script (e.g., study design(s))
test data matches scope & passes "FDA Data Fit" assessment (?)
documentation inputs/outputs/dependencies/usage
Proposal through CSS
2104
• Heavy qualification
– Beta package includes
•
•
•
•
minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
– Draft
• Write qualification plan, Review tests for completeness/suitability
(e.g., Branch testing – are all conditional blocks/combos tested?)
– Test
• Peer Review: Write qualification report, incl. log/output from tests
– Final
Proposal through CSS
2104
• Light qualification
– Beta package includes
– Draft
•
•
•
•
•
skip if >1 yr production use without ERROR
minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Write qualification plan, Review tests for completeness/suitability (e.g.,
Branch testing – are all conditional blocks/combos tested?)
– Test
• Peer Review: Write qualification report, incl. log/output from tests
– Final
Proposal through CSS
2104
Peer Review Checklist
Heavy
Light
Requirement specification
X
?
Documented or perhaps only documented in header
X
User Guide
X
X
SDTM/ADaM used in input/output
X
X
Open CDISC validator or Data Fit used to check input/output
X
X
GPP in source
X
X
Run according to Requirement specification
X
?
Tested by qualification plan, tests & results all Peer reviewed
X
?
Tested by End users
X
?
Robust without red errors in contributor's production environment
X
X
Robust and used in FDA (other) scripts repository, ranked ******
X
Main Sections
• Summary of prior proposal
Updated proposal
o Motivation & objectives, as justification for
elements of proposal
Proposal 2014
Motivation
• End-user Objectives
–
–
–
–
–
–
Clear overview of resources available, and the purpose & state of each
Inspire confidence from first user experience
Ease of script use, clear messaging from first run of scripts
Reproducible results in user's own environment
Consistency of scripts, learning first one makes remaining familiar
Ease of converting users to contributors
• Contributor & Team Objectives
– Clear, standardized workflows and checklists
– Modularize routine components (e.g., FUTS for dependency
checking?)
– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)
– Centralize & consolidate information & results
Qualification Proposal
meaningful terms in blue
http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal__2014.pptx
• Qualification Instructions (see embedded template )
– Certification phase of Qualification applies to new scripts and tests
– Confirmation phase applies to updates of existing scripts
• States:
• Roles
Contributed, Development, Testing, Qualified
–
–
–
–
Contributor: Anyone with appropriate skills & interests
Developer: CSS Working Group 5 volunteer familiar with objectives**
Tester: CSS WG 05 volunteer familiar with objectives**
Environment Tester: Anyone in industry community able to set up
automatic test replication in their work environment
– Reviewer: Author of white papers, designers of script targets**
** suggests a quick-start onboarding page in CSS Phusewiki
Proposal
Qualification
• Metadata for script should indicate
–
–
–
–
–
–
Whitepaper ID & output ID
Programming language & version (e.g., R v3.1.1, SAS v9.4)
Type of output:
tabulated data, analysis data, table, figure, listing
Study design:
parallel, crossover, etc
State of qualification:
Contributed, Development, Testing, Qualified
OS is not included, since should be indicated in OS compatibility report
• Test Data requirements
–
–
–
–
available as a program or script (text, not binary)
based on expected standards (SDTM, ADaM)
data requirements clearly & accurately specified for each script
include expected results from these data for defined tests/checks
Proposal
Qualification
• Transitions
"Contributed" is the original State of all scripts
– to Development, checklist includes
by Developer & Reviewer
• R & D confer on suitability of contribution. Suitable starting point?
[ may require analysis details, specs, from contributor ]
• D reviews components (checklist to be completed)
• D works with Contributor to complete minimum components
[ including Test Data and Coverage of defined tests ]
• D adds standard parameter, dependency checking
• D writes Qualification instructions .docx (see template, above)
– to Testing, checklist includes
• Review Qualification instructions, consider coverage of tests
• Execute Qualification instructions
• Work with Developer to complete execution successfully
by Tester
Proposal
Qualification
• Transitions
– to Qualified
•
•
•
•
continued
by Tester & Environment Tester & Reviewer
T updates reference test outputs from certification/confirmation
E updates & executes local tests (posting PASS/FAIL results)
R confirms script output matches intention
R confirms qualification process covers important elements and
considerations.
• R also provides user (rather than technical) feedback?
• Achieve "Qualified" state when all tests in all test environments
PASS (i.e., match outputs that T has certified and/or confirmed)
and that R agrees that target is achieved
Proposal
Qualification
•
Efforts Required
– Top priority
• Finalize Qualification states, roles, workflow, checklists, and templates
– Next priorities
• Design test structure in google code
• Develop scripts that will allow Environment Testing
• Develop general components (e.g. parameter, dependency checking)
• Identify Environment Testers based on
– Host environment
– SAS or R version
• Identify opportunities to automate qualification. E.g.,
– Environment Testers to post results back as machine readable
– Script green-light/red-light qualification matrix on Phusewiki