Transcript Developmental Test and Evaluation Committee Status and Plans February 2014 Beth Wilson, Raytheon Steve Scukanec, Northrop Grumman Industry Co-Chairs NDIA SE Division Meeting February 12, 2014

Developmental Test and Evaluation
Committee
Status and Plans
February 2014
Beth Wilson, Raytheon
Steve Scukanec, Northrop Grumman
Industry Co-Chairs
NDIA SE Division Meeting
February 12, 2014
1
DT&E Committee
2014 Status and Plans
SE
T&E
Activity
Plans for 2014
Status
Cyber Testing
Industry recommendations for
cyber testing
Draft complete
Submit to DASD DT&E Feb
Chief Developmental
Tester
Industry interaction with new role
White paper outline complete
Soliciting ase study examples
Reliability
Recommendations for better
reliability testing
IEEE standards liaisons
established
T&E Workshop
Summer workshop
Tutorials and papers
Modeling & Simulation
T&E Perspective for Modeling/
Simulation products
Joint meeting TBD
Architecture
T&E Perspective for Architecture
views
Joint meeting TBD
Education & Training
Chief Developmental Tester role
(after CDT project complete)
Systems Security
Engineering
Cyber testing guidelines
connections to PPP
(after SSE PPP May workshop)
SE Conference
October conference
DT&E track, joint tracks
NDIA SE Division Meeting
February 12, 2014
2
Complete
In Process
Proposed
Cyber Testing Guidelines
Activity
Cyber Testing
Plans for 2013 - 2014
Review what industry does for
cyber testing, expand focus to
include interoperability
Status/Plans
Recommendations for Cyber Testing
guidelines
Project Lead: Dave Desjardins
Joint Project with ITEA
NDIA SE Division Meeting
February 12, 2014
3
Chief Developmental Tester
Activity
Chief Developmental
Tester
Plans for 2013-2014
Status/Plans
Review service policies for new
role and industry
implementations
Propose model for industry interaction
White paper
T&E/SE conference presentations
The Secretary of Defense shall require that each major defense
acquisition program be supported by—
‘‘(A) a chief developmental tester; and
‘‘(B) a governmental test agency, serving as lead developmental
test and evaluation organization for the program.
Coordinating DT&E activities
Insight into Contractor activities
Oversee T&E activities
Inform government PM about contractor DT&E results
Project Lead:
Joe Manas
NDIA SE Division Meeting
February 12, 2014
4
Reliability Testing
Activity
Reliability Testing
(with IEEE)
Plans for 2013
Develop a better test strategy
for reliability (collaborate with
IEEE Reliability Society)
Status/Plans
Solicit feedback on existing standards
Identify recommendations to improve
reliability testing
Reliability Testing Focus
- Software reliability standard
- Cyber testing initiatives
- Data for reliability growth measures
Input/Feedback Opportunities:
- IEEE meetings January and August
- Periodic revision periods
Project Leads:
Lou Gullo, Taz Daughtrey
NDIA SE Division Meeting
February 12, 2014
5
DT&E Committee - 2014 Task Plan
Projects Working Group
Proposed 2014 Tasks:
Deliverables/Products
•
•
•
•
• Cyber Testing Recommendations white paper
• Chief Developmental Tester white paper
• Candidate KSABs for role (E&T)
• Industry interaction recommendation
• Tutorial on role for SE conference
• Inputs to reliability standards effort
• Joint meeting notes
Cyber Testing Project
Chief Developmental Tester Project
Collaboration with IEEE Reliability
Joint meetings with SED Committees
• Modeling and Simulation
• Architecture
• Systems Security Engineering
• Education and Training
Schedule / Resources
Issues / Concerns:
• Cyber Testing Draft Recommendations Feb 2014
• Weekly teleconferences started 12/3/2013
• Full white paper later in 2014
• Chief Developmental Tester
• Joint meeting with E&T with SED meeting
• Tutorial at SE conference
• Reliability standards input for Jan/Aug meetings
• Test as a Stakeholder joint meetings with SED
• Modeling and Simulation
• Architecture
• Systems Security Engineering
• Industry engagement
• NDIA T&E conference may be cancelled (Seattle
in March)
• ITEA conference (Denver in September) may be
postponed and moved
NDIA SE Division Meeting
February 12, 2014
6
Backup
DT&E Committee Results 2008-2013
NDIA SE Division Meeting
February 12, 2014
7
DT&E Committee Current Structure
Since 2010
NDIA
Systems Engineering
Division
NDIA
Test and Evaluation
Division
SED
Committee
Collaborative
Projects
T&E
Themes
For
Conferences
NDIA Industrial Committee
on Test and Evaluation
(ICOTE)
NDIA
DT&E Committee
SE Themes & Issues
Project Results
DT&E Themes
NDIA Systems Engineering Conference (Q4)
Collaborative Projects
T&E Themes & Issues
NDIA Test and Evaluation Annual Conference (Q1)
Periodic T&E Committee-Focused Conferences
Cooperation with other NDIA Divisions
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
NDIA SE Division Meeting
February 12, 2014
8
Summary of DT&E Committee
Test and Evaluation Efforts
DT&E Committee
Topic
Activity
DoD T&E Policy Study
2006 – 2008: Workshops and Study Report
Improving T&E in the DoD Acquisition Process
Integrated Testing
2008 – 2010: Integrated Test Study
NDIA presentations and tutorial
ITEA journal article
RFP Language for T&E
2010 – 2011: Comments for Update to OSD Guide
Incorporating T&E into DoD Acquisition Contracts
Test Optimization
2012: Statistical Optimization Conference Thread
2013: Scientific Test and Analysis Techniques report/articles
Cyber Testing
Started 2013: Industry recommendations for cyber testing
Chief Developmental Tester
Started 2013: Industry interaction with new role
NDIA SE Division Meeting
February 12, 2014
9
Summary of DT&E Committee
Systems Engineering Efforts
SE Collaboration
SED Committee
Activity
Software
2009: SW T&E Summit Recommendations
System of Systems
2010 – 2011: T&E for SoS Workshop and Initiatives
2012: Final Report on Best Practices Model
Performance
Measurement
2012: Leading Indicators for T&E workshop
Modeling and
Simulation
2011: Effective Use of Modeling and Simulation for T&E
2012: Distributed Model-Based Testing
Reliability
Started 2013: Recommendations for better reliability testing
NDIA SE Division Meeting
February 12, 2014
2013: Requirements Verification leading indicators presentation
10
DoD T&E Policy Study
August 2006: DT&E Committee Kickoff
Policy Study:
“Improving T&E in the DoD Acquisition Process”
Industry T&E policy recommendations
Workshops:
August 2007
January 2008
Focus Areas:
1. Earlier contractor and tester involvement
2. Integrated DT/OT and DT operational relevance
3. Suitability
April 2008: Report Summarized Results:
10 Findings
15 Recommendations
2006-2008
NDIA SE Division Meeting
February 12, 2014
11
Integrated Testing (CT/DT/OT)
Implementation Framework
2008-2010
NDIA SE Division Meeting
February 12, 2014
12
Software Test and Evaluation
Software Summit
RFP Language
Training & Competency Model
Policy, Guidance, and Standards
2009
Tools, Automation, Methodologies, Process
NDIA SE Division Meeting
February 12, 2014
How Much
Testing
is
Enough?
Lifecycle
and
End-to-End
Software T&E
Changing
Paradigms
13
RFP Language
Master #
Industry #
Reviewer
Line
Comment and Rationale Recommended Input A/R/P
#
Industry Comments for Update:
Incorporating Test and Evaluation
Into Department of Defense
Acquisition Contracts
Recommendations
from SW Summit
2010-2011
NDIA SE Division Meeting
February 12, 2014
14
Test and Evaluation for
Systems of Systems
2009: “Sleepless Nights”
List of Issues
2010: “Sominex”
Resulting Initiatives
2010: Workshop
2011: Best Practices
Wave Model
External Environment
Initiate
SoS
Conduct
SoS Analysis
Evolve
SoS
Arch
Develop
SoS
Arch
Implement
SoS
Update
NDIA SE Division Meeting
February 12, 2014
Continue
SoS Analysis
Evolve
SoS
Arch
Plan
SoS
Update
Plan
SoS
Update
2009-2012
Continue
SoS Analysis
Continue
SoS Analysis
Plan
SoS
Update
Implement
SoS
Update
Implement
SoS
Update
2012: Final Report
15
Effective Use of Modeling and
Simulation for Test and Evaluation
Joint Meeting in August 2011
• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test
and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data
Management Working Group
• LVC Architecture Roadmap Implementation
(LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group
• Raytheon Presentation on M&S for T&E
• Potential Topics for November AMSWG Meeting
2011
NDIA SE Division Meeting
February 12, 2014
16
Modeling & Simulation for
Distributed Testing
Benefits:
•Find integration issues earlier
•Test to learn in ‘safe’ environment
•Protect proprietary information
•Facilitate DT to OT transition
•Increase performance testing range in operating environments
•Support end to end studies throughout the program
Barriers:
•Security
•Lack of persistent network
•Early consideration of technical issues
•Perceived value
•Disconnect between the communities (M&S and T&E)
Recommendations:
•Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)
•Create a framework for reusing and repurposing M&S through the product model
•Establish M&S as part of statistical test design
•Determine what tests are conducted to acquire data for model validation.
•Fewer test events with better models.
•Recommend the use of M&S to do I&T
•Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint Meeting
August 2012
Joint Track
SE Conference October 2012
2012
NDIA SE Division Meeting
February 12, 2014
17
Statistical Test Optimization
2012 SE Conference summit/workshop thread
-Tutorials on Monday 10/22
- Presentations on Wednesday 10/24
- Synthesis Panel on Wednesday 10/24
Follow-on Results:
- DT&E Committee white paper
- ITEA Journal article
- SE Conference track Oct 2013
- MORS Conference Nov 2013
2012 - 2013
NDIA SE Division Meeting
February 12, 2014
18
Requirements Verification
Leading Indicator Metrics
2012 SE Conference workshop
- Follow-on workshop with System Performance Measurement WG
- Focused on Requirements Verification not addressed in first report
Potential Measures Identified:
- System Maturity Level
- Verification Requirement Maturity
- Technical Measures and Stakeholder Need
2012 - 2013
NDIA SE Division Meeting
February 12, 2014
19