Transcript Software Build and Packaging Techniques

Introduction to
Software Build Technology
Philip Johnson
Collaborative Software Development Laboratory
Information and Computer Sciences
University of Hawaii
Honolulu HI 96822
(1)
Objectives
Understand motivation for build and packaging
technologies.
• Why was running your classmate’s Robocode
robot so hard?
Become familiar with a sample of build
technologies.
Get introduced to the build technologies and
standards to be used in this class.
(2)
Motivation
Modern team-based, cross-platform development involves:
• Multiple developers and users.
• System is developed/used on multiple platforms:
- Win2K, Linux, Win98, Solaris
• Developers/users have different environments:
- c:\jdk1.5.0_08, d:\jdk1.5, c:\myjava
• Differences between user and developer distributions:
- Source distribution for developers
- Binary distribution for users
• Developers may prefer different IDEs:
- Eclipse, JBuilder, Emacs, RationalRose
• Users may not have any IDE at all.
• Frequent releases and distributions.
• Dependence on third party libraries (which may themselves be
released frequently).
These issues are not unique to Java development!
(3)
Motivation (cont.)
Build/packaging techniques should allow:
• Users to build and test the system without IDE.
• Developers to build and test the system using their IDE of
choice.
• Developers to build and test the way users build and test.
• Configuration to different platforms and different
environments within a platform.
• Multiple release configurations
• Automated build to prevent errors
• Rapid release, deployment, testing
• Dependency management for 3rd party libraries
(4)
A quick tour of build tools
(5)
(6)
Make
‘Make’ is the canonical build tool.
Make features:
• File-level dependency analysis
- Example: compile only changed files
• Unix-based (but ported to other platforms)
• Language independent (Java, C++, etc.)
• ASCII configuration file syntax
- Tab characters required—Yikes!
• Supports invocation of any shell command.
• Does not support library-level dependency management.
• Related: imake, cmake, scons
(7)
(8)
Ant
Java-based build tool by Apache
• http://ant.apache.org/
Features:
• Java-based
• Cross-platform
• Extensible
• XML configuration files
• Open source
• Dependency management provided by Ivy.
(9)
(10)
Maven
“Project Management” technology
• Build based on Ant.
• Documentation and reporting plugins.
• File-level dependency management
• Library-level dependency management
• Library repository definition & management
“Convention over configuration”
• Maven builds in conventions for project layout
and workflow to simplify use.
-Great if you like them.
(11)
Ant and Maven
Both based upon XML configuration files.
Both provide a high-level "Domain Specific
Language" for building systems.
• DSL makes assumptions about what you want
to do and how you want to do it.
Extensible via "plugins"
• More difficult to write
(12)
Rake
(13)
Rake
"Thin coating" of build functionality on top of
Ruby language:
• Predefined libraries for packaging.
• Prerequisites for dependencies.
Not a DSL:
• If you know Ruby, you're good to go.
(14)
Buildr
(15)
Buildr
A "rake for Java"
• Underlying language is Ruby
• Understands Java files and Maven file
structure
(16)
rake and buildr
Do not attempt to provide a DSL for builds.
Requires you to (eventually) learn a general
purpose programming language (Ruby).
No "plugins" required---just write code for your
special situations.
(17)
The basic divide
Domain-specific language
• Ant
• Maven
Syntactic Sugar
• Make
• Rake
• Buildr
Read "Why everyone (eventually) hates (or leaves)
Maven for a good discussion of the divide:
DSLs get you farther, faster, initially
Ultimately, the DSL gets in your way as your needs get more
sophisticated.
(18)
Our approach
We will use Maven
• You get "farther, faster" using Maven.
• Many projects use it.
• Our projects do not have (many)
idiosyncracies that make Maven life difficult.
(19)
Basic Ant concepts
Every project requires an Ant build file
• named build.xml by default.
Each build file is composed of targets.
• Typical targets: compile, junit, etc.
Each target is composed of tasks
• Ex: copy .java files to build/src, invoke javac...
Targets can have dependencies
• Ex: modified sources must be recompiled before
testing.
(20)
Ant targets
Dependent targets are executed exactly once.
• Example:
-test depends upon compile
-deploy depends upon compile
-all depends upon test, deploy
• Compile will be executed only once.
Some targets are executed only when needed.
• Sources recompiled only if changed.
(21)
Example Ant target
<target name="compile"
depends=“init"
description="Compiles code.">
<javac srcdir="${src.dir}"
destdir="${build.dir}/classes“/>
</target>
(22)
Best practices
Ant provides a simple, powerful language for
building Java-based systems.
How do we use this language effectively?
The next section describes my best approach
so far on how to employ Ant to achieve the
Three Prime Directives of Open Source
Software Engineering.
The “robocode-pmj-dacruzer” example project
provides you with a template structure to get
you off and running quickly.
(23)
robocode-pmj-dacruzer:
An example Ant-based build system
Implements a pretty lame Robocode robot.
• Not very interesting.
It’s the build system that’s interesting:
• Provides a template
- Easy to adapt to your own projects.
• Integrates third-party build tools
- Checkstyle, PMD, FindBugs, etc.
• Is modular and extensible
- You can grow or shrink it to your needs.
• Well suited to build automation and project hosting:
- Continuous integration (Hudson)
• Uses Ivy to download/install/manage third-party libraries.
Result is Maven-ish, but without the “black box”
• If you want to change something, it’s just Ant code.
(24)
Top-level directory contents
build.xml
• top-level build file.
*.build.xml
• “module” build files, generally one per tool.
src/
• the system source files
lib/
• library files: maintained by Ivy.
build/
• derived files: generated by build process
(25)
Build operations
compile
• compile system
jar
• create jar file
junit
• run junit tests
javadoc
• build javadoc docs
clean
• delete build dir.
dist
• create zip dist file.
(26)
checkstyle
• analyze source code
findbugs
• analyze byte code
pmd
• analyze source code
sclc
• compute size of system
jacoco
• compute coverage of test
cases.
(Others to be added later in
semester!)
Build System Scalability Issues
Modularity:
• putting all targets in one build.xml file creates a
very large build.xml file that is difficult to
understand and maintain.
Incrementality:
• you may not want to require all developers to
have to install every possible third party tool.
Coupling:
• dependencies between target and property
definitions should be minimal and easily visible.
(27)
Build system architecture
build.xml contains
most projectspecific
definitions.
build.xml
pmd.build.xml
findbugs.build.xml
dist.build.xml
checkstyle.build.xml
javadoc.build.xml
sclc.build.xml
common.build.xml
boilerplate code
(28)
*.build.xml
*.build.xml files
import the
build.xml file
and provide an
implementation
of a single
extension.
build.xml
build.xml provides most (but unfortunately not all) of the
project-specific definitions:
• version number
• dependency libraries
• project name
• etc.
Most other *.build.xml files can be used with little to no
changes.
• Typical exceptions: junit.build.xml, emma.build.xml,
jar.build.xml
(29)
*.build.xml responsibilities
Implements support for a single tool/function:
• Checkstyle, Jar, JavaDoc, PMD, Distribution, etc.
Typical tasks (assume a tool called 'foo'):
• Download foo using Ivy, install in lib/foo
• Download a configuration file for foo, store in
lib/configfiles.
• Provide three standard tasks:
- foo.tool: runs foo tool, generates data.
- foo.report: creates HTML file.
- foo: runs foo.tool and foo.report.
• Make 'foo' the default task.
(30)
User Interface
To do initial installation and compilation:
• ant
For other operations, use -f <filename>:
• ant -f dist.build.xml
• ant -f junit.build.xml
• ant -f checkstyle.build.xml
• ant -f pmd.build.xml
• ant -f emma.build.xml
• etc.
(31)
Considerations
This approach provides the following:
• Minimal coupling: *.build.xml files depend
only on definitions in build.xml
• Modularity: individual *.build.xml files are
small.
• Reporting: "report" targets provide details on
what the "tool" targets generated.
One problem:
• How to "verify" that all analyses "passed"?
(32)
verify.build.xml
Provides “one stop shop” for system analysis:
• Imports not just build.xml, but other
*.build.xml files as well.
• Runs all "tool" analyses, failing build if they
generate any warnings/errors.
-Compile, Junit, PMD, Checkstyle, JavaDoc,
FindBugs
User interface:
• ant -f verify.build.xml
(33)
The lib/ directory
Contains third-party libraries required by
project:
• robocode
• checkstyle
• pmd
• findbugs
• etc.
Contains configuration files for analysis tools.
• lib/configfiles/checkstyle.modules.xml
• lib/configfiles/pmd.rulesets.xml
• lib/configfiles/findbugs.filter.xml
(34)
The build/ directory
Contains files derived from build process.
Can be deleted and reconstructed.
Not included in version control or distribution .zip.
Build subdirs:
• build/javadoc/
• build/classes/
• build/dist/
• build/sclc/
• build/junit/
(35)
Targets creating them:
javadoc
compile
dist
size
junit
build/ vs. bin/
Both Ant and Eclipse (or any other IDE)
compiles files.
It causes problems if Ant and Eclipse are using
the same output directories for compilation/etc.
Therefore, Eclipse writes to the bin/ directory,
not the build/ directory.
• Set in Project | Properties | Java Build Path |
Source
(36)
Testing Notes
There are two kinds of “tests” in this system:
• Manual QA using JUnit
-Ensures your code satisfies its specifications (if
you write good test cases.)
• Automated QA using Checkstyle, FindBugs, PMD
-Ensures your code obeys best practices for
coding style.
Encourages incremental testing and formatting.
(37)
Traditional release numbering
Traditional approach (i.e. jakarta-tomcat-5.1.0.zip)
• MM.mm.bb, where:
- MM = Major release number
(major redesign or incompatible changes)
- mm = Minor release number
(minor enhancements, backward compatible)
- bb = Bugfix release number
(bug fix, no new functionality)
• All numbers maintained manually.
Advantage:
• Release number comparison indicates degree of change.
Disadvantage:
• Requires manual editing of release value every time which leads to
errors (easy to forget to do this.)
(38)
Our release numbering
Our approach (i.e. robocode-pmj-dacruzer-1.3.2011.09.11.12.32.zip)
• Major.minor.timestamp, where:
- Major = major release number
- Minor= minor release number
- Timestamp= time of bugfix release (yyyy.mm.dd.hh.mm)
• Major and minor release number maintained manually.
• Bugfix timestamp generated automatically.
Advantages:
• Automatic except when incrementing major/minor release.
• Release number provides timestamp of release.
Disadvantages:
• Number of bugfix releases not indicated by release number.
• Very long version number.
(39)
IDE integration
Important to maintain independence from IDE.
• All build functions should be possible without
any IDE.
But it’s fine to provide IDE-specific files to
facilitate development.
• Example: .class and .project files in top-level
directory for Eclipse.
Provide additional files/directories for other
IDEs if desired.
(40)
robocode-pmj-dacruzer
as a standard
From now on, all projects must be developed,
packaged, and submitted using the standard
build conventions embodied in this package.
(41)
robocode-pmj-dacruzer
as a template
You can use robocode-pmj-dacruzer as a template
for your own projects:
• Download a fresh version.
• Remove dacruzer-specific info
• Add your project info.
See the template instruction page in the course
website for step-by-step instructions.
(42)
Ant/Eclipse integration
Eclipse has nice support for Ant, but:
• You will also need to tell Eclipse about ANT_HOME:
-Window | Preferences | Ant | Runtime | Ant Home…
• Make sure Eclipse writes to the bin/ directory
-Project | Properties | Java Build Path | Source
(43)
Limitations of example
The robocode-pmj-dacruzer template provides a
framework plus basic build functionality, but
has limitations:
• No SVN integration
• No support for web app development.
• No support for automated metrics collection.
We will enhance this initial template during the
semester.
(44)
Meta-level goal
Ant provides an efficient solution to satisfying
both
• PD#2 (users can install system) and
• PD#3 (developers can understand and
enhance system)
Other approaches exist:
• InstallAnywhere for PD#2 (more work)
• IDE-specific solution for PD#3 (limited)
With good installation documentation, I believe
Ant-based build can satisfy both PD#2 and
PD#3.
(45)
(46)