Transcript Developmental Test and Evaluation Committee Beth Wilson, Raytheon Steve Scukanec, Northrop Grumman Industry Co-Chairs February 2013 NDIA SE Division Meeting February 13, 2013

Developmental Test and Evaluation
Committee
Beth Wilson, Raytheon
Steve Scukanec, Northrop Grumman
Industry Co-Chairs
February 2013
NDIA SE Division Meeting
February 13, 2013
1
DT&E Committee Focus
2013
Activity
Plans for 2013
Status/Plans
Scientific Test and
Analysis Techniques
STAT for T&E
Examples of effective use of
statistical approaches for test
optimization (including DOE) for
an implementation framework
Status: draft white paper review
Thread at T&E conference (cancelled)
Plans: participate in May MORS
conference, results at SE conference
Modeling &Simulation
Test as a Stakeholder
Focus on test as stakeholder for
M&S products (collaboration on
M&S lifecycle sub-team project)
Collaboration with M&S Committee
Joint meeting: TBD
(possibly M&S, Architecture, DT&E)
Architecture
Test as a Stakeholder
Develop T&E perspective for
architecture efforts and
products
Collaboration with Architecture WG
Effort TBD
Potential joint meeting with M&S
NDIA T&E Conf
DT&E panel, STAT report
Conference cancelled
NDIA SE Conf
DT&E tracks
Joint M&S, STAT report
Added Task:
Metrics
Provide more information on
leading indicators for T&E
Collaboration with System
Performance Measurement WG
Added Task:
Reliability
Develop a better test strategy
for reliability (collaborate with
IEEE Reliability Society)
Effort TBD
NDIA SE Division Meeting
February 13, 2013
2
DT&E Committee - 2013 Task Plan
Projects Working Group
Proposed 2013 Tasks:
• Scientific Test and Analysis Techniques
(including DOE)
• Joint Meeting with Modeling and Simulation
• Test as a Stakeholder for Architecture
Additional 2013 Tasks:
• Better test strategy for reliability (collaborate
with IEEE Reliability Society)
• Provide more information on proposed
leading indicators
Deliverables/Products
• Scientific Test and Analysis Techniques:
• White paper
• Thread at T&E conference
• Results at SE conference
• Joint Meetings with SED committees
• Modeling and Simulation
• Architecture
Schedule / Resources
Issues / Concerns:
• STAT White Paper: weekly progress tag-up
• Conference threads:
• T&E February 2013 [CANCELLED]
• SE October 2013
• Joint meetings: TBD
• Resource availability
• Resources:
• DT&E committee members
• STAT white paper sub-committee
NDIA SE Division Meeting
February 13, 2013
3
DT&E
Statistical Test Optimization
Activity
Scientific Test and
Analysis Techniques
STAT for T&E
Plans for 2013
Status/Plans
Examples of effective use of
statistical approaches for test
optimization (including DOE) for
an implementation framework
Status: draft white paper review
Thread at T&E conference (cancelled)
Plans: participate in May MORS
conference, results at SE conference
2012 SE Conference summit/workshop thread
-Tutorials on Monday 10/22
- Presentations on Wednesday 10/24
- Synthesis Panel on Wednesday 10/24
Define an implementation framework
- Define attributes of successful application
- Identify framework for implementation
- Provide amalgamation of success examples
NDIA SE Division Meeting
February 13, 2013
T&E Conference
CANCELLED
February 2013
SE Conference
Report
October 2013
4
NDIA DT&E Committee
Summary of Accomplishments
2006 – 2012
NDIA SE Division Meeting
February 13, 2013
5
DT&E Committee Current Structure
Since 2010
OSD
SE
Issues
Identified
White
Papers
NDIA
Systems
Engineering
Division
SED
Committee
Collaborative
Projects
NDIA
Test and
Evaluation
Division
T&E
Themes
For
Conferences
NDIA
DT&E Committee
Project Results
DT&E Themes
NDIA
Industrial
Committee
on
Test
and
Evaluation
(ICOTE)
Issues
Identified
OSD
OT&E/
LFT&E
Discussion
OSD
DT&E
White
Papers,
Journal
Articles &
Workshops
T&E Themes & Issues
NDIA Systems Engineering
Conference (Q4),
Collaborative Projects
NDIA Test and Evaluation
Annual Conference (Q1), Periodic T&E Committee-Focused
Conferences, Cooperation with other NDIA Divisions
DT&E Committee Focus: T&E initiatives aligned with SE, DT&E
NDIA SE Division Meeting
February 13, 2013
6
Summary of
DT&E Committee Efforts
SE Collaboration
DT&E Committee
Topic
Activity
DoD T&E Policy Study
2006 – 2008: Workshops and Study Report
Improving T&E in the DoD Acquisition Process
Integrated Testing
2008 – 2010: Integrated Test Study
NDIA presentations and tutorial
ITEA journal article
RFP Language for T&E
2010 – 2011: Comments for Update to OSD Guide
Incorporating T&E into DoD Acquisition Contracts
Test Optimization
2012: Statistical Optimization Conference Thread
Software
2009: SW T&E Summit Recommendations
System of Systems
2010 – 2011: T&E for SoS Workshop and Initiatives
2012: Final Report on Best Practices Model
Modeling and Simulation
2011: Effective Use of M&S for T&E Use of M&S
2012: Distributed Model-Based Testing
Metrics
2012: Leading Indicators for T&E
Architecture
Future: T&E Perspective for Architecture views
NDIA SE Division Meeting
February 13, 2013
7
DoD T&E Policy Study
August 2006: DT&E Committee Kickoff
Policy Study:
“Improving T&E in the DoD Acquisition Process”
Industry T&E policy recommendations
Workshops:
August 2007
January 2008
Focus Areas:
1. Earlier contractor and tester involvement
2. Integrated DT/OT and DT operational relevance
3. Suitability
April 2008: Report Summarized Results:
10 Findings
15 Recommendations
2006-2008
NDIA SE Division Meeting
February 13, 2013
8
Integrated Testing (CT/DT/OT)
Implementation Framework
2008-2010
NDIA SE Division Meeting
February 13, 2013
9
Software Test and Evaluation
Software Summit
RFP Language
Training & Competency Model
Policy, Guidance, and Standards
2009
Tools, Automation, Methodologies, Process
NDIA SE Division Meeting
February 13, 2013
How Much
Testing
is
Enough?
Lifecycle
and
End-to-End
Software T&E
Changing
Paradigms
10
RFP Language
Master #
Industry #
Reviewer
Line
Comment and Rationale Recommended Input A/R/P
#
Industry Comments for Update:
Incorporating Test and Evaluation
Into Department of Defense
Acquisition Contracts
Recommendations
from SW Summit
2010-2011
NDIA SE Division Meeting
February 13, 2013
11
Test and Evaluation for
Systems of Systems
2009: “Sleepless Nights”
List of Issues
2010: “Sominex”
Resulting Initiatives
2010: Workshop
2011: Best Practices
Wave Model
External Environment
Initiate
SoS
Conduct
SoS Analysis
Evolve
SoS
Arch
Develop
SoS
Arch
Implement
SoS
Update
NDIA SE Division Meeting
February 13, 2013
Continue
SoS Analysis
Evolve
SoS
Arch
Plan
SoS
Update
Plan
SoS
Update
2009-2012
Continue
SoS Analysis
Continue
SoS Analysis
Plan
SoS
Update
Implement
SoS
Update
Implement
SoS
Update
2012: Final Report
12
Effective Use of Modeling and
Simulation for Test and Evaluation
Joint Meeting in August 2011
• Distributed Testing, the Joint Mission
Environment Test Capability (JMETC) and the Test
and Training Enabling Architecture (TENA)
• DoD M&S Community of Interest Data
Management Working Group
• LVC Architecture Roadmap Implementation
(LVCAR‐I) Gateways Effort Applicability to T&E
• OSD T&E Working Group
• Raytheon Presentation on M&S for T&E
• Potential Topics for November AMSWG Meeting
2011
NDIA SE Division Meeting
February 13, 2013
13
Modeling & Simulation Collaboration
Distributed Testing
Benefits:
•Find integration issues earlier
•Test to learn in ‘safe’ environment
•Protect proprietary information
•Facilitate DT to OT transition
•Increase performance testing range in operating environments
•Support end to end studies throughout the program
Barriers:
•Security
•Lack of persistent network
•Early consideration of technical issues
•Perceived value
•Disconnect between the communities (M&S and T&E)
Recommendations:
•Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)
•Create a framework for reusing and repurposing M&S through the product model
•Establish M&S as part of statistical test design
•Determine what tests are conducted to acquire data for model validation.
•Fewer test events with better models.
•Recommend the use of M&S to do I&T
•Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T
Joint Meeting
August 2012
Joint Track
SE Conference October 2012
2012
NDIA SE Division Meeting
February 13, 2013
14
Metrics Collaboration
Leading Indicators for T&E
Workshop October 22, 2012
Follow-on workshop with System Performance Measurement WG
Focused on other information needs not addressed in first report
http://www.ndia.org/Divisions/Divisions/SystemsEngineering/Documents/Studies/NDIA%20System%20Develpopment%20Performance%20Measurement%20Report.pdf
2012
NDIA SE Division Meeting
February 13, 2013
15