Transcript Systems Engineering Effectiveness Committee Activity Report NDIA SE Division Meeting 18 August 2010 SE Effectiveness Committee.

Systems Engineering
Effectiveness Committee
Activity Report
NDIA SE Division Meeting
18 August 2010
SE Effectiveness Committee
1
Systems Engineering Effectiveness Activity Since Last SE Division Meeting
Form a subcommittee:
•
to develop a research proposal for the next phase of the SE Effectiveness
study. (Mike McLendon / Marvin Ebbert /Joe Elm)
•
made up of IEEE and NDIA representatives to discuss SE Effectiveness
study extension. (Joe Elm / Bob Lyons)
•
to plan a workshop on system development performance measures
focused on readiness to proceed at milestones. (Garry Roedler, Chris
Miller)
•
to organize a study and/or workshop on “Why good programs succeed and
why bad programs fail”. (Al Brown, Chris Miller, Dave Griffith, Geoff Draper)
•
to identify and collect available information on SEP and SEMP
effectiveness. (Dona Lee, Bob Scheurer)
SE Effectiveness Committee
2
SE Effectiveness Study Extension (Business
Case for Systems Engineering) – Status
Situation:
• At the request of OSD, on May 6 we briefed Mr. Stephen Welby (DDR&E Director Systems Engineering) on the SE
Effectiveness Study results
-
Study completed back in 2007 but it never gained much traction
Mr. Welby agreed that the info was good and he would be a conduit for making it known to DoD.
Mr. Welby saw the value of having data on more domains but also realized that if he did nothing to recognized the value of the
work to date, we could not expect to get the additional data that he desires.
We need a solid and reasonable study plan with options to begin dialogue with OSD on next steps.
Mike McLendon, CTR OSD ATL, agreed to be engaged on this effort
Action:
• Set-up Working Group with IEEE to expand SE Effectiveness study to a wider group
Augmentation of action:
• Develop a research proposal detailing the next steps
-
In collaboration with IEEE and the DoD, expand and improve upon the NDIA SE Effectiveness Study, using survey to identify
SE activities that show net benefit to program performance.
Develop recommendations to OSD for policy and guidance updates (if required) to implement the findings of the expanded
survey.
Establish a means of structuring acquisition program reporting to facilitate the on-going collection of data on SE activity as a
mechanism for continuous improvement
Action Status
• Working group formed with the IEEE Aerospace & Electronic Systems Society (AES) to reach a far broader audience.
-
Also, Mr. Dan Goddard, Chief of AF Division, AFMC/EN, Mr. Clarence Gooden, Civ USAF SAF/AQRE, and Mr Marty Meth,
former pentagon official, have joined the team
• Holding weekly LiveMeeting/Telecon
-
Reviewing NDIA SE Effectiveness survey documentation
Reviewing survey questions for suggested refinements / additions
Drafting e-mail to encourage NDIA and IEEE member participation
Developed a draft of research proposal for future work
Plan to report on progress at Annual SE Conference in San Diego
SE Effectiveness Committee
3
System Development Performance
Measurement – Status
Situation:
• WSARA required joint DT&E and SE tracking and reporting on MDAP achievement of measurable performance criteria
- Goal is to enable objective, information-based decision making
- Bring better measurement of performance criteria into Program Support Reviews (PSRs)
Action:
• Plan a workshop on system development performance measures focused on readiness to proceed at milestones. This
workshop should include OSD and the Services.
Augmentation of action:
• Establish a plan to form a Working Group under the SEEC to address system development performance measures as
required by WSARA and needed by OSD.
-
The plan should reflect a similar process as used for the Development Planning WG.
As part of the plan include the planning of a workshop focused on identifying information needs and potential measures.
The working group should include collaboration with PSM and participation from OSD and the Services.
Action Status:
• Garry Roedler and Chris Miller developed a briefing on the planning and actions for establishing a special working
group under the SEEC and preparation needed for a workshop on System Development Performance
Measurement. The plan includes:
-
Formation of a Core Team
Analysis of industry resources
Development of a position paper
Invitations to and formation of the Working Group
Planning and conduct of the Workshop
Post-workshop actions – briefing OSD leadership, action plan, and working group report
• Collaboration with Practical Software and Systems Measurement (PSM) confirmed.
• Briefing on the planning and actions to be reviewed at the SEEC meeting, August 18
SE Effectiveness Committee
4
“Why Good Programs Succeed and why bad
Programs Fail” – Status
Situation:
• Strong interest in critical success and failure characteristics for programs
-
Propose NDIA SEEC do cross industry analysis on critical success/failure characteristics – work in conjunction with NDIA PM
Committee
– Leverage existing studies as basis and analyze for common threads (some from NDIA SED)
• Systemic Root Cause Analysis; SE Effectiveness Survey; GAO Studies; Army study (McGarry); NDIA
Gulf Coast Chapter Study; and Other industry studies
• Consider full life cycle, especially pre-M/S A
– Develop prioritized list of critical characteristics
– Develop a set of recommendations
Action:
• To organize a study and/or workshop on “Why good programs succeed and why bad programs fail”.
Augmentation of action:
• The biggest concern is that over the past several years this question/task has been given to several different groups
(SERC, MITRE, SE Effectiveness, Systematic Database, …). They all established achievable objectives and each one
resulted in a credible product that met those objectives. Yet in the eyes of AT&L leadership none of them were ‘silver
bullet’ solutions and therefore they were never embraced. The concern is not whether or not OSD (Jim Thompson)
‘wants’ this report, but one of skepticism that what is wanted is feasible. Chris Miller has agreed to confirm this task
request with Jim, but the primary issue is that we don’t want our team to commit to a study that is dead on arrival (even
before we begin). The subcommittee plans to discuss more at a side-bar during the August 18th meeting.
Action Status:
• This action is on hold, pending further confirmation/guidance from Jim Thompson and discussion by the subcommittee.
SE Effectiveness Committee
5
SEP and SEMP Effectiveness – Status
Situation:
• Colonel Shawn P. Shanley, SAF/AQRE Director, Engineering and Technical Management Division, suggested the
committee look into the effectiveness of SEPs and SEMPs
Action:
• Identify and collect available information on SEP and SEMP effectiveness
Augmentation of action:
• Dona Lee agree to investigate the background of this action item resulting from the April SE Division meeting to see
what was intended as well as contact Col Shawn Shanley to see what the AF is doing in this area and report back to
see whether there is anything further to do in this area.
Action Status:
• Dona contacted Col Shanley and he sent her the Air Force SEP study but it is closehold. Dona has reviewed the
document for discussion in our August 18 committee meeting.
• In a parallel effort, Bob Scheurer has developed a short draft questionnaire for soliciting SEP – SEMP development
and utilization from NDIA industry members. It will be designed to ensure that responses are completely anonymous.
SE Effectiveness Committee
6
NDIA SE Conference 2010 Technical Program
Committee, Abstract Review – Status
Situation:
• As a primary Committee Chair of the NDIA Systems Engineering Division, and/or a past Track Chair at the NDIA
Systems Engineering Conference, Dona Lee and Al Brown are members of the Systems Engineering Conference 2010
Technical Program Committee (TPC), representing the Systems Engineering Effectiveness Committee.
Action:
• NDIA SE Conference Paper Selection for the Systems Engineering Effectiveness track
Augmentation of action:
• Provide the briefing order for the papers/panels assigned to you track.
• Added a SE Standards Panel (=3 papers) because we think standards represent best practices and there is a lot going
on now referencing bringing some back
• Prioritized order of presentation of papers for the track.
Action Status:
• Complete
SE Effectiveness Committee
7