Transcript Continuous integration and test

CONTINUOUS
INTEGRATION AND TEST
Stephen Oglesby - CSCI577b – April 2011
Introduction
2






Who am I?
What is Continuous Integration?
Continuous Integration + Automated Testing
Requirements for successful CI&T
Scaling options
CI&T within CSCI577ab
“Cake, and grief counseling, will be available at the end of the testing period.” - GLaDOS
4/27/2011
Continuous Integration
3


“Continuous Integration is a software development
practice where members of a team integrate their
work frequently, usually each person integrates at
least daily - leading to multiple integrations per day.
Each integration is verified by an automated build
(including test) to detect integration errors as quickly
as possible.” – Martin Fowler
Born out of eXtreme Programming (XP), not limited
to it.
4/27/2011
What does that mean?
4





Code commit detection
Per check-in compilation (yes you should do this
anyway, but now it’s automatic)
Automated build solution
Expandable to include static code analysis
Results reported to the team
 Immediate
notification of integration errors
 Faster resolution
4/27/2011
CI & Test
5



Automated unit & acceptance testing is a natural
extension of CI
Reduces test time and cost
Improve communication & project visibility
 Updated
and available test results throughout
development
 Bug Tracking System integration for automatically
logging new defects

Reduce risk
4/27/2011
Requirements
6




Single source repository
Automated build
Automated self-test
Tests run on production environment clone
4/27/2011
Requirements (cont.)
7

Fast builds
4/27/2011
Scaling Up from XP
8

Add Hierarchy
 Execute
build at each level of integration
 Label version units based on integration test depth
Copper
Silver
Copper
Gold
Copper
Silver
Copper

Add Builds
 Stable,
Beta, Dev, Nightlies
4/27/2011
CI&T in CSCI577ab
9





Requires bringing test activities into the first
semester
Test case definition during requirements
gathering/WikiWinWin
Still requires test case dependencies, even more
critical since automation environment needs them
Standardized test procedures and results,
automatically generated
Automation Framework + CI Server should be
largely reusable across most projects
4/27/2011
CI&T in CSCI577ab (cont.)
10

Provides quantitative assessment criteria
 Percentage
of development time with a failed build
 Number of overnight build failures
 Number of deleted test cases
 Percentage of requirements with corresponding test
cases
 Percentage of methods not tested

Increases team collaboration, code quality,
developer productivity
4/27/2011
Open Source CI&T Solutions
11

Automated Test Frameworks
 Framework
for Integrated Test (Fit + Fitnesse)
 Software Testing Automation Framework (STAF)
 Selenium

Continuous Integration Server
 CruiseControl
+ Subversion
“At the end of the experiment you will be baked, and then there will be cake” - GLaDOS
4/27/2011
References
12








Sean Stolberg, “Enabling Agile Testing Through Continuous Integration”, Agile Conference 2009, IEEE, pg
369-374, 2009
Hehui Liu, Zhongjie Li, Jun Zhu, Huafang Tan, and Heyuan Huang, “A Unified Test Framework for Continuous
Integration Testing of SOA Solutions”, 2009 International Conference on Web Services, IEEE, pg 880-887,
2009
Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Test Automation Framework for Implementing Continuous
Integration”, Sixth International Conference on Information Technology: New Generations, IEEE, pg 784-789,
2009
Eun Ha Kim, Jong Chae Na, and Seok Moon Ryoo, “Implementing an Effective Test Automation Framework”,
23rd Annual IEEE International Computer Software and Applications Conference, IEEE, pg 534-538, 2009
Jon Bowyer and Janet Hughes, “Assessing Undergraduate Experience of Continuous Integration and TestDriven Development”, 28th International Conference on Software Engineering, ACM, pg 691-694, 2006
Fabrizio Cannizzo, Robbie Clutton, and Raghav Ramesh, “Pushing the Boundaries of Testing and Continuous
Integration”, Agile 2008 Conference, IEEE, pg 501-505, 2008
Christopher G. Jones, “Test-Driven Development Goes to School”, Journal of Computing Sciences in Colleges,
Volume 20, Issue 1, pg 220-231, 2004
Martin Fowler, “Continuous Integration”, http://www.martinfowler.com/articles/continuousIntegration.html,
2006
4/27/2011
13
Questions?
“Well done. Here are the test results: You are a horrible person. I'm serious, that's what it
says: A horrible person. We weren't even testing for that.” - GLaDOS
4/27/2011