Transcript Title

Standard Script All-Hands meeting
September 29, 2014
1
PhUSE CSS Background
• Computational Science Symposium (CSS) Working
Groups
– 5th year in a Series of Annual Computational Science
Meetings established to support the work of FDA’s Center
for Drug Evaluation and Research (CDER), Office of
Computational Science (OSC)
– 3rd Year working with PhUSE (first two were with the DIA)
– FDA’s motivation: To work collaboratively with industry,
vendors and regulators in a “non-competitive space” to
find solutions and advance the computational
sciences/technology associated with the development and
regulation of new medical products (drugs, biologics, etc.)
– Perceived benefits of collaboration and crowd sourcing
2
PhUSE CSS Working Groups
Optimizing Use of
Data Standards
Standard Scripts for
Analysis and
Reporting
Emerging
Technologies
Semantic
Technologies
Non-Clinical
White Papers on
Analysis and Displays
Script Repository (4
projects)
Communications
3
Summary of Standardization
Clinical
Data
Flow
Trial
Design
Data
Collection
Systems
Observed
Datasets
Analysis
Datasets
Tables,
Figures
and
Listings
CDASH
SDTM
ADaM
No TFL Stds
Exist
Industry
Standards
PRM
Alignment
?
TransCelerate (TAs)
Script Repository
Projects:
• P01: Look for existing scripts and store them in the repository - Adrienne
Bonwick and Aiyu Li
• P02: Define qualification steps for scripts in the repository - Dante Di
Tommaso
• P03: Maintain and enhance platform (repository) for sharing scripts - Mike
Carniello/Hanming Tu
• P04: Legal ownership and issues in open source repository - Sally Cassells
• P05: Create templates and metadata for documenting scripts and coding
practices - Jean-Marc Ferran/Eric Sun
• P06: Refine process for creating and editing scripts in Google Code
• P07: Implement and further develop communication plan for standard
scripts - Dirk Spruck
• P08: Create white papers providing recommended display and analysis
including Table, List and Figure shells - Mary Nilsson
P01:
Adrienne Bonwick and Aiyu Li
Remit:
• Look for existing scripts and store them in the repository
Progress
• Received the “Demographics” package from the FDA
• Reviewed and updated to work on any PC SAS package using FDA Pilot
Data
• Provided working instructions
• Added to Code Repository
• Starting to investigate MAED
Standard Scripts Project 2
2014 - Proposal for Qualification of
Standard Scripts
Main Sections
Summary of prior proposal, 2013
Updated proposal, July 2014
Main Sections
Summary of prior proposal
– Concepts, definitions & metadata
– Test data considerations
– Heavy vs. Light qualification
Updated proposal
Proposal from 2013
• Anyone can submit a script, according to a check list
• Categorize scripts by complexity
– Complexity:
– Output:
low, medium, high, software
tabulated data, analysis data, table, figure, listing
• Metadata for script should indicate
– Type of output: tabulated data, analysis data, table, figure, listing
– Study design:
parallel, crossover, etc
– State of qualification:
?
Proposal from 2013
• Test data
– Overall project should have minimum test data (SDTM & ADaM)
– Scripts can propose new test data, must pass (Data fit? Open CDISC?)
– Share program to produce test data, never binary test data
• 2 levels of qualification
– Light vs. Heavy: according to complexity of script & output
– Common elements include
•
•
•
•
•
header
adhere to good programming practices
clear scope of script (e.g., study design(s))
test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?)
documentation available for: usage, inputs, outputs, dependencies
Proposal from 2013
• Heavy qualification
– Beta package includes:
•
•
•
•
minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC?)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
– Draft
• Write qualification plan, Review tests for completeness/suitability (e.g.,
Branch testing – are all conditional blocks/combos tested?)
– Test
• Peer Review: Write qualification report, incl. log/output from tests
– Final
Proposal from 2013
• Light qualification
– Beta package includes
– Draft
•
•
•
•
•
skip if >1 yr production use without ERROR
minimal elements for contribution
Specification & Documentation (could be in pgm header)
Test data (Data Fit? or Open CDISC or other, as appropriate)
Tests & Expected results defined
Peer Review: GPP, Specs & Docn reviewed, Tests reproduced
Write qualification plan, Review tests for completeness/suitability (e.g.,
Branch testing – are all conditional blocks/combos tested?)
– Test
• Peer Review: Write qualification report, incl. log/output from tests
– Final
Proposal through CSS 2104
Peer Review Checklist
Heavy
Light
Requirement specification
X
?
Documented or perhaps only documented in header
X
User Guide
X
X
SDTM/ADaM used in input/output
X
X
Open CDISC validator or Data Fit used to check input/output
X
X
GPP in source
X
X
Run according to Requirement specification
X
?
Tested by qualification plan, tests & results all Peer reviewed
X
?
Tested by End users
X
?
Robust without red errors in contributor's production environment
X
X
Robust and used in FDA (other) scripts repository, ranked ******
X
Main Sections
Summary of prior proposal
Updated proposal
– Objectives
– Definitions: Qualification, States, Roles
– Metadata and Test data
– State Transitions
Proposal 2014 - Objectives
• For End-users
– Clear overview of resources available, incl. purpose & state of each
– Inspire confidence from first user experience
– Ease-of-use:
• clear messaging from scripts
• Reproducible results in user's own environment
• Consistency of scripts, learning first one makes remaining familiar
– Ease of converting users to contributors
• For Contributors & Standard Scripts Team
–
–
–
–
Standardize workflows and checklists
Modularize routine components (e.g., FUTS for dependency checking?)
Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)
Centralize & consolidate information & results
Qualification Proposal
meaningful terms in blue
http://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal__2014.pptx
• Qualification Instructions (see embedded template )
– Certification phase of Qualification applies to new scripts and tests
– Confirmation phase applies to updates of existing scripts
• States:
• Roles
Contributed, Development, Testing, Qualified
–
–
–
–
Contributor: Anyone with appropriate skills & interests
Developer: CSS Working Group 5 volunteer familiar with objectives**
Tester: CSS WG 05 volunteer familiar with objectives**
Environment Tester: Anyone in industry community able to set up automatic
test replication in their work environment
– Reviewer: Author of white papers, designers of script targets**
** suggests a quick-start onboarding page in CSS Phusewiki
Qualification Proposal
• Metadata for scripts should indicate:
–
–
–
–
–
–
YML proposal
Whitepaper & output ID uniquely identify the target <Script: Source, Title, Target>
Programming language & version
<Language: name, version>
Type of output:
<Script: Type>
Study design:
<Script: StudyDesign>
State of qualification <Stage>:
<Qualification: Stage>
OS is not included, since should be indicated in OS compatibility report
• Test Data requirements
–
–
–
–
available as a program or script (text, not binary)
based on expected standards (SDTM, ADaM)
data requirements clearly & accurately specified for each script
include expected results from these data for defined tests/checks
Qualification Proposal
• Transitions
"Contributed" is the original State of all scripts
– to Development, checklist includes
by Developer & Reviewer
• R & D confer on suitability of contribution. Suitable starting point?
[ may require analysis details, specs, from contributor ]
• D reviews components (checklist to be completed)
• D works with Contributor to complete minimum components
[ including Test Data and Coverage of defined tests ]
• D adds standard parameter, dependency checking
• D writes Qualification instructions .docx (see template, above)
– to Testing, checklist includes
• Review Qualification instructions, consider coverage of tests
• Execute Qualification instructions
• Work with Developer to complete execution successfully
by Tester
Qualification Proposal
• Transitions
– to Qualified
•
•
•
•
continued
by Tester & Environment Tester & Reviewer
T updates reference test outputs from certification/confirmation
E updates & executes local tests (posting PASS/FAIL results)
R confirms script output matches intention
R confirms qualification process covers important elements and
considerations.
• R also provides user (rather than technical) feedback?
• Achieve "Qualified" state when all tests in all test environments
PASS (i.e., match outputs that T has certified and/or confirmed)
and that R agrees that target is achieved
Qualification Proposal
• Efforts Required
– Top priority
• Finalize Qualification states, roles, workflow, checklists, and templates
– Next priorities
• Design test structure in google code
• Develop scripts that will allow Environment Testing
• Develop general components (e.g. parameter, dependency checking)
• Identify Environment Testers based on
– Host environment
– SAS or R version
• Identify opportunities to automate qualification. E.g.,
– Environment Testers to post results back as machine readable
– Script green-light/red-light qualification matrix on Phusewiki
P03:
Hanming Tu and Mike Carniello
Remit:
• Maintain and enhance script repository
Progress
• CSS 2014 Scriptathon work catalogued and displayed (see next slide)
• PharmaSUG Scriptathon work to be added to repository
• PhUSE 2014 Scriptathon planned
What’s next?
• Do we use Scriptathons as vehicle for adding code?
• Should we spend effort in updating/perfecting what we have?
• Scriptathon targets are White Paper displays … should we add SDTM and
ADaM scripts as well?
• https://code.google.com/p/phuse-scripts/wiki/yml_idx
P04: Legal ownership and issues in the Open
Source Repository
Summary
16 Sep 2014
Background
• Members of WG5, the PhUSE Standard Script
Development group, wanted to ensure that
users of posted scripts are protected from any
potential claims of ownership from
organizations employing individuals who
contribute scripts.
– The group recommended that a ‘click through’
agreement be obtained from a lawyer and posted
on the GoogleCode site. This became P04 for our
group.
Assignment March 2013
• Get more scripts in the Script Repository
– Networking
– Broad Communications
• Address any remaining validation, legal, process
issues
• Finalize first white paper
– Goal to finalize within a couple of months
• Begin work on the remaining white papers
– Looking for more active participants
– Goal to be final or near-final by next year
Summary of Next Steps (2013)
• The work has now been divided into 8 projects
– P01: Look for existing scripts and store them in the
repository (Kevin Kane)
– P02: Define validation steps for scripts in the
repository (Lina Jørgensen)
– P03: Maintain and enhance platform (repository)
for sharing scripts (Hanming Tu)
– P04: Legal ownership and issues in open source
repository (Kevin Kane/Sally Cassells)
Summary of Next Steps (2013)
• 8 projects (cont’d)
– P05: Create templates for documenting scripts
and coding practices (Jean-Marc Ferran)
– P06: Refine process for creating and editing scripts
in Google Code (Kevin Kane)
– P07: Implement and further develop
communication plan for standard scripts (Dirk
Spruck)
– P08: Create white papers providing recommended
display and analysis including Table, List and
Figure shells (Mary Nilsson)
Actions for P04
• Sally contacted Chris Decker to request that
PhUSE provide a legal contact who could
develop the agreement.
• Chris provided draft agreement in May 2013
for review.
• The agreement is currently posted on the
GoogleCode site and in the PhUSE Wiki . The
document name is SS 15591982 1 UKGROUPS
Harmony Individual Contributor License
AgreementDOC.
Issues, Discussions, Decisions
• UK vs. US
– The Draft Agreement cites UK law rather than US
(Section 8). Adding similar clauses for the US would
require the services of a US lawyer.
• Individually owned tools
– Contributor grants full ’worldwide, non exclusive,
royalty free, irrevocable license to the IP’.
– If tool was developed collaboratively, contributor
should get agreement from collaborators before
posting.
• Click through
– Not yet technically click through but contributors can
read and agree to the terms.
Next Steps, Recommendations
• Investigate possibility of modifying the
language in Section 8 to indicate that the
agreement applies fully within the US.
• Consider a leadership team vote to finalize the
draft agreement.
• Link to agreement PhUSE Standard Scripts
Contributor Agreement
P08: Mary Nilsson
Project Description
• Create white papers providing recommended displays and
analyses including Table, Figure, and Listing shells for clinical
trial study reports and summary documents.
• The intent is to begin the process of developing industry
standards with respect to analysis and reporting for
measurements that are common across clinical trials and
across therapeutic areas.
• Script developers could then create scripts consistent with the
recommendations for all to use, improving efficiency and
safety signal detection.
P08: Mary Nilsson
Progress - The following white papers have been finalized:
• Analyses and Displays Associated with Measures of Central Tendency –
With a Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and
Integrated Summary Documents (finalized in October 2013)
• Analyses and Displays Associated with Non-Compartmental
Pharmacokinetics – With a Focus on Clinical Trials (finalized in March
2014)
Note: Final white papers can be found on the PhUSE website, Publications
section. (www.phuse.eu)
P08: Mary Nilsson
Progress - The following white papers are in progress:
•
Analyses and Displays Associated with Outliers or Shifts from Normal to Abnormal – With a
Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and Integrated Summary
Documents (third draft targeted for public comment in September 2014)
• Analyses and Displays Associated with Adverse Events – With a Focus on Phase 2-4 Clinical
Trials and Integrated Summary Documents (first draft targeted for public comment in
September 2014)
• Analyses and Displays Associated with Demographics, Disposition, and Medications – With a
Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (final white paper
targeted for October 2014)
• Analyses and Displays Associated with Hepatotoxicity – With a Focus on Phase 2-4 Clinical
Trials and Integrated Summary Documents (first draft targeted for public comment by the end
of 2014)
• Analyses and Displays Associated with QT Studies (first draft targeted for public comment by
the end of 2014)
Note: Draft white papers (where applicable) can be found on the PhUSE Wiki, Development of
Standard Scripts Project 08 wiki page (www.phusewiki.org)
P08: Mary Nilsson
• How you can help
– When a white paper has been finalized, or when a
white paper is ready for broad review, please help
spread the word within your organization and
across your contacts. Please include your medical
colleagues. Thanks!