Transcript Middleware Progress report and plans

Wojciech Sliwinski
[email protected]
Beams Department, Controls Group
CERN
Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
2
Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
3
Software Quality (SWQ)…

We know it’s “a good thing” ™

People think…
 “I know it when I see it”
 “I know the value of it”
But what are the actual characteristics of SWQ?
 Do we have a common understanding of SWQ?


The lack of it normally appears as consequences…
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
4
1985: Therac-25 – Software issues…

Radiation Deaths linked to Software Errors
 Killed 2 people
 Injured dozens of others
 http://www.ccnr.org/fatal_dose.html
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
5
1996: Ariane 5 – floating point conversion error
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
6
2007: Airbus A340 – Software issues …
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
7
80s – The Kano Model (Marketing)
Customer Satisfaction
Satisfied
Needs well
fulfilled
Needs not
fulfilled
Dissatisfied
Product Features
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
8
Early and Other Quality Definitions

Zen and the art of motorcycle maintenance, Robert Pirsig,
1974:
 “If quality exists in an object, then you must explain why scientific
instruments are unable to detect it”
 “On the other hand, if quality is subjective, [existing only in the eye
of the observer] then this Quality is just a fancy name for whatever
you'd like.”
 “Quality is not objective. It doesn't reside in the material world [...]
Quality is not subjective. It doesn't reside merely in the mind.”

McCall et al, 1977 and Boehm et al, 1978
First elaborate studies on the notion of 'software quality’
 Quality factors – only indirectly measurable like “reliability”
 Quality criteria – objective aspects that support the factors, like
“uptime”.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
9
IEEE SWQ Standards

IEEE Glossary of Software Engineering Terminology
defines it as
 “the degree to which a system, component, or process meets
customer or user needs or expectations”

IEEE Std 1061-1992 IEEE Standard for a Software Quality
Metrics Methodology
 Does not actually prescribe any metrics!
 Defines method to evaluate metrics that could be applicable to
a particular case

Plenty more – eg. IEEE Std 730 and friends
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
10
ISO Quality Standards

ISO 9000 series of standards for quality management
systems:
 Process based
 “the degree to which a set of inherent characteristics fulfils the
requirements”

ISO 9126 Software Quality Characteristics and Metrics,
2001
 Comprehensive but … large & complex
 6 key quality attributes:
○ Functionality, Reliability, Usability
○ Efficiency, Maintainability, Portability
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
11
SWQ – Huge Set of Characteristics
• Only a subset applies to all projects
• Choose those that suit the project goals
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
12
McCalls’ categories of SWQ factors
Operation:
Correctness
Does it do what I want?
Reliability
Does it do it accurately all of the time?
Efficiency
Will it run on my hardware as well as it can?
Integrity
Is it secure?
Usability
Can I run it?
Revision:
Maintainability
Can I fix it?
Testability
Can I test it?
Flexibility
Can I change it?
Transition:
Portability
Will I be able to use it on another machine?
Reusability
Will I be able to reuse some of the software?
Interoperability
Will I be able to interface it with another system?
Source: J.A. McCall, P.K. Richards and G.F. Walters, Factors in Software Quality,
RADC-TR-77-369, 1977, US Dep. of Commerce.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
13
Internal & External Quality Attributes
Important Primarily to Users
Important Primarily to Developers
Availability
Maintainability
Efficiency
Portability
Flexibility
Reusability
Integrity
Testability
Interoperability
Reliability
Robustness
Usability
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
14
The SW Quality “Iceberg”
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
15
The Maintainability Index (MI)
MI = 171 – 5.2 x ln(aveV)
- 0.23 x aveV(g’)
- 16.2 x ln(aveLOC)
+ 50 x sin √2.4PerCM
Parameter
Name
Measures
aveV
Average Halstead complexity
Computational density
aveV (g′)
Average extended cyclomatic complexity
Logical complexity
aveLOC
Average count of lines of code
Code size
PerCM
Average percent of lines of comments
Human insight
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
16
MI applied to FreeBSD codebase
Kernel Codebase
User Programs
Modules with MI < 40 are rejected !
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
17
Applying Quality

Computers don’t care about quality!


Metrics are tests for quality characteristics
Quality “clusters”

Testing is ensuring a quality of conformance
 Bugs occur in clusters too!

Like testing, if quality is low, builds should fail

Quality characteristics must be part of the whole
development lifecycle
 Quality products only come from quality requirements, quality
design, quality code, …
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
18
Applying Quality (Assurance)
Software Quality consists of:
1.
2.
3.
4.
5.
6.
7.
Engineering Methods (proper analysis, design, …)
Standards and Procedures (common to all)
Technical Reviews (eg. code reviews)
Configuration management (repositories, versioning)
Measurement (code, metrics)
Testing (unit, system, integration, acceptance)
Improvement (of the items above)
• Quality is a way of life
• If you think Quality is expensive, try an accident…
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
19
Capability Maturity Model (CMM) Levels*
1.
Initial (chaotic, ad hoc, individual heroics)
○
2.
Repeatable
○
3.
the process is defined/confirmed as a standard business process, and
decomposed to levels 1 and 2
Managed (quantatively)
○
5.
some processes are repeatable, possibly with consistent results
Defined
○
4.
the starting point for use of a new process.
using process metrics, management can identify ways to adjust and adapt
the process to particular projects
Optimized
○
process management includes deliberate process
optimization/improvement.
* Capability Maturity Model from SE Institute.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
20
Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
21
Continuous Improvement (CI) – Origins
Deming’s Plan-Do-Check-Act (PDCA)
 The Toyota Production System

 “Stop the line” quality control





About eliminating “waste” (obstacles)
A learning, non-judgmental whole team approach
Adopted by the Agile movement for SWD
Caveat: Development is an “empirical process”
Introduced when an organization recognizes it has
arrived at some impasse
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
22
CI - How it works

“People (at all levels) freely discuss the information,
issues, ideas, evaluate them, choose, plan and execute
improvements”

Focuses on how things get done:
 Standardized (and ing) common tasks
 Only fixing possible issues
 e.g.: repeated mistakes, time sinks, quality holes

Very suitable where tools, technology, external
outcomes and requirements change often
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
23
CI - What it relies on

Objective information:
 To evaluate current situation and processes
 Assess applied improvements
 Judge outcome as worthwhile, or retry
 Demonstrate added value of CI against overhead

Improvement culture:
 “Free speech” – everyone’s ideas are welcome
 A real desire to try it and improve
 Accepting there will not be total agreement
 Replacing skeptical inaction with real experimentation 
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
24
CI – It can fail!
No one “owns” responsibility for implementation
 No charter to make changes
 No time allocated for improvement
 No follow up by team or sponsors

24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
25
Improvement Practices

Change Agents
 AKA: Coaches, Monitors, Consultants, Champions
 One or more floating people searching for improvements

Kaizen time aka “renovation days”
 Team plans some improvements
 All actions executed by all at same time

Reflection workshops
 Regular meetings to decide on improvements
 Apply one or more ideas
 Keep what works
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
26
CI - Summary






Team driven incremental changes
Team evolves and improves together
Drives increase in quality, efficiency and satisfaction
Not a tool or technique – it is an attitude
Not leader oriented, instead peer based consensus
Keeps us ahead in best practice and professionalism
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
27
Bibliography

Software Requirements
 K. Wiegers, Microsoft Press, 2003

Code Complete: A Practical Handbook of Software Construction
 S. McConnell, Microsoft Press, 2004

Software Engineering: Principles and Practice
 H. van Vliet, John Wiley & Sons, 2008

Wikipedia page on SWQ ISO standard 9126.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
28
Bibliography
The Toyota Way, J. K. Liker, 2004
 Agile and Iterative Development, C. Larman, 2004
 Sustainable Software Development, Kevin Tate, 2006
 Implementing Lean Software Development, M. & T. Poppendick,
2007
 Rapid Development, S. McConnell, 1996
 The Enterprise and Scrum, K. Schwaber, 2007
 Collaboration Explained, J. Tabaka, 2006
 Peopleware Papers: The Human side of Software, L. Constantine,
2001

24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
29
Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
30
Context - CERN Accelerator Complex
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
31
Context - The CERN Control Centre
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
32
Controls Software Infrastructure
PUBLIC
ETHERNET
NETWORK
OPERATOR
CONSOLES
FIXED
DISPLAYS
Presentation Layer
A 3-tier architecture
OPERATOR
CONSOLES
•
Client
tier
FILE
SERVERS
APPLICATI
ON
SERVERS
SCADA
SERVE
RS
Business Layer
CTRL CTRL
RT Lynx/OS
VME Front Ends
WORLDFIP
Front Ends
T
CMW
T
T
PLC
T
BEAM POSITION MONITORS,
BEAM LOSS MONITORS,
BEAM INTERLOCKS,
RF SYSTEMS, ETC…
T
T
QUENCH PROTECTION AGENTS,
POWER CONVERTERS FUNCTIONS
GENERATORS, …
PROFIBU
S
T
WorldFIP
SEGMENT
(1, 2.5 MBits/sec)
OPTICAL
FIBERS
T
T
•
TCP/IP communication
services
Device Layer
Resource
tier
•
TCP/IP communication
services
TIMING
GENERATION
FIP/IO
CERN GIGABIT ETHERNET
TECHNICAL NETWORK
Server
tier
DB
TCP/IP communication
services
ACTUATORS AND SENSORS
CRYOGENICS, VACUUM,
ETC…
•
Presentation Layer
– Graphical interactive
applications
– Fixed Displays
Business Layer
– General purpose and
specific Application
servers
Device Layer
– Front End
Device servers
Communication to the
equipment goes through
Controls MiddleWare CMW
LHC MACHINE
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
33
Controls Software Codebase - Situation

Complex Accelerator domain

~6 million LOC
Java/J2EE

 ~200 developers (CERN + other labs)
 ~1000 projects/modules
Control System
GUI Applications

 ~35 developers (CERN + other labs)
 +20 major projects
Control Logic
Middleware






24th October 2011
C/C++
Oracle database
Eclipse IDE + plugins
SVN + Maven + CommonBuild(
Atlassian Suite (JIRA, Fisheye, Crucible, etc.)
Nowadays encourage Scrum
Focused on Software Quality
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
34
Controls Software Codebase - Situation

Lots of Code
 ~6 million lines, 20’000+ classes.
 … and still growing
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
35
Situation – Self Assessment and Outlook

Unknown quality
 Quality unchecked
 No mentoring or guidance?
 Limited standards

High Staff turnover
 Students, Project Associates, Fellows …
More projects coming (PS renovation, …)
 More service provision (looking after running services)
 Will this cause future problems like maintenance?

24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
36
Outline

Defining Software Quality

Continuous Improvement

Context – Accelerator Controls Codebase

Applying QA & CI for the Controls Codebase

Conclusions
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
37
Objectives – improve Software Quality
 Introduce quality improvement as an integral
part of the everyday development work
 Leverage tools to automate the process as much as
possible
 Establish guidelines and metrics to measure progress
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
38
Quality in the Development Cycle

For each stage
 Agreed mandatory activities and deliverables
 Tools identified and integrated with IDE (Eclipse), giving
immediate feedback
Design
Deploy,
Maintain
Implement,
Test,
Document
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
39
Design Reviews




Design
Design meetings with
external members
To verify the soundness of
design in an early stage
To identify overlapping
functionality
Promote a set of common
components and libraries
24th October 2011
Deploy,
Maintain
Implement,
Test,
Document


At the level of subcomponents, main classes
and design patterns
UML: class and sequence
diagrams
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
40
Code Reviews & Code Analysis
Design
Deploy,
Maintain
Implement,
Test,
Document


Goal:
 Identify defects
Person-to-person time
consuming
 Ensure maintainable code
 Only for critical code
 Verify conventions are followed
 Mentoring of junior

Static code analysis tools to
identify common mistakes and
bug patterns:
 FindBugs
developers

Light-weight person-toperson code reviews with
FishEye + Crucible
 Checkstyle
 Eclipse warnings
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
41
Author
and
reviewers
Comments
inline
Files being
reviewed
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
42
Reviewing Code (Crucible)
43
The ‘bug’
A line
list of
indicated
‘bugs’
‘Bug’
explained
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
44
Testing & Continuous Integration
Design
Deploy,
Maintain
Implement,
Test,
Document


General agreement on benefits
of testing and CI
To increase level of testing, unit
tests mandatory deliverable of
each project
 >30% test coverage for non-trivial
classes, measured with Clover
24th October 2011

To discover inter-project
issues early, Continuous
Builds with Bamboo:
 CI server from Atlassian
 Compiles and runs unit tests
 Builds fail on:
○ Low test coverage
○ Basic PMD rules
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
45
Continuous Integration and Test (Bamboo)
Test coverage for project
Code metrics
Classes with highest risk
46
Red =
not covered
Green=
covered
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
47
Top/flop list generated
(Work in progress)
accsoftcommonscore
Triggered by
changes in a
dependency
accsoftcommon
s-util
accsoftcommonsconcentration
24th October 2011
accsoftcommonsio
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
48
Integration and System Testing with CO Testbed

CO Testbed - mini-accelerator lab
 Hardware and Software
 Core components from the Accelerators Controls duplicated
into this Test Environment

System and Integration tests are executed by Bamboo CI
server
 Automatic execution of tests with “real” system
 Automatic deployment of Release Candidates
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
49
TIMING
FEC03
FEC01
FEC05
24th October 2011
SERVER06
SERVER07
FEC02
FEC04
Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN
50
Integration Tests done through Client APIs
Bamboo CI Server
TESTS
(JAPC, CMW, RBAC)
JMS
CMW Proxy
CMW, RBAC
A2, FESA
CMW, RBAC
A2, FESA
PPC/LynxOS
Linux/i386
24th October 2011
RBAC
A1 Service
CMW
Directory Svc
Timing
Config
DB
Wojciech Sliwinski: CI and QA for the Accelerator
Controls Codebase at CERN
Client APIs
51
CO Testbed controlled by Bamboo CI server
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
52
1.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
53
2.
1.
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
54
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
55
Challenges
Agree on standards and configurations
Think of and organize us as one big team, not many
small ones
 The code belongs to the group, not to a project or an
individual developer
 The guidelines and standards established and agreed
by everyone
Challenges

24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
56
Encourage developers
to invest time on quality
Quality-related tasks part of the yearly objectives for
a project
 Progress measured against the level a project
adheres to guidelines – top/flop lists
 Focus the effort where the effect is highest, rely on
tools as much as possible
 ‘CI days’ – common days with a common goal
Challenges

24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
57
Conclusions

Integrated quality improvement in the development cycle

Introduced guidelines/standards and supporting tools

Increased developer motivation through
 Immediate feedback when developing
 Official tracking of progress and top/flop lists

Increased awareness of benefits, application in individual
projects and confidence when making changes

Agile approach  reflect, identify ”waste” and improve
24th October 2011
Wojciech Sliwinski: CI and QA for the Accelerator Controls Codebase at CERN
58