Transcript RedHat_Nam_Duong_1-1.2

Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Red Hat on Jenkins: By the
numbers
Nam Duong
Red Hat, Inc.
https://openshift.redhat.com
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Agenda
Products currently leveraging Jenkins
Deployment size: # of Masters/Slaves/Jobs in use
across Jboss Middleware QE
Top 10 list of interesting Plugins
Code contributions
Experiences/Pitfalls/Lessons learned
Best Practices
Demo:
Production/Staging server (sheer numbers)
OpenShift on Jenkins
EC2 and GIT plugin demo
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Red Hat products
Products currently leveraging Jenkins
JBoss offerings:
Enterprise Application Platform
JBoss Developer Studio
SOA
JBoss Portal
JBoss OSGI
Many, many others
OpenShift – OpenShift is our Platform as a
Service (PaaS) offering. Our Jenkins instances
are running in the cloud, hosted on OpenShift
Flex and used to build and test OpenShift
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Deployment Size
JBoss Middleware Team:
– 1 Master
• IBM System x3550 M3, 2x E5640 CPU, 24GB RAM, 2x
SAS 73GB 15K-RPM in RAID1
– Slaves connected 113 (various OS: linux, Win, Solaris,
HP, AIX)
– ~4700 Jobs configured (~1000 running/month)
– ~95% of Middleware test automation is
called/executed via Jenkins
OpenShift Flex Team:
– 1 Master
• 7.5 GB, 4CPU, 10GB EBS (RHEL 5.6)
– 10 Slaves Building EC2 Images and 5 slaves running
tests
– 20 jobs (RHEL 6.1)
– ~90% of our automation is executed via Jenkins
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Interesting plug-ins used
Management (Groovy*, Disk Usage*, Build
Publisher*, Workspace Cleanup*)
Cloud Control (EC2 plugin, DeltaCloud*)
Enterprise focused plugins (SmartFrog)
Code Coverage (cobertura, emma)
Source control (GIT/SVN/CVS)
*Represents code contributions
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Code Contributions
Memory leaks: Involved in diagnosis, resolution
In the Hudson timeframe, we contributed to
stability through reporting of bugs/instability
(large deployment)
First ones to run into these types of problems
(pure volume)
LTS (long term support - Jenkins stable branch)
Development and maintenance of plugins (see
previous)
reporting bug + bug fixes
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Experiences/Pitfalls/Lessons learned
1 master may not handle all jobs at once potentially provide failover functionality on
Jenkins
Jobs getting stuck due to polling console output
through ssh. Need to limit jobs from writing to
console.
Disk Utilization issues - Overcome through quotas
set on jobs
Logging was too verbose, Jenkins instance stalled
due to low/no disk space
OOM issues when trying to view the giant logs - fix
in progress
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Best Practices
Set limits on memory usage (quota) of slave
Set limits (quota) on Disk usage for jobs
Tools on shared NFS drive – for maintenance
and replication/recovery
Memory – lots of memory to buffer
incremental growth of Master
Jenkins master and all slaves under CFEngine
control (config management/recovery)
Like all reliable services, use Dev/Stg/Prod
increments.
Can scale vertically, but has anyone tried
scaling horizontally?
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Demo
Production instance within corporate firewall:
Production: <Screenshots available upon request>
Staging our plugins
Staging: <Screenshots available upon request>
Jenkins running on OpenShift Flex; testing
OpenShift Flex
<Screenshots available upon request> EC2 and GIT plugin
demo
Used to create a client testbed
Creates an EC2 instance, prints the IP address, pulls down
all code bits
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Links/Contact
Email: [email protected]
https://openshift.redhat.com
We appreciate your feedback of your user experience
Jenkins User Conference
San Francisco, Oct 2nd 2011
#jenkinsconf
Thank You To Our Sponsors
Platinum
Sponsor
Gold
Sponsor
Silver
Sponsor
Bronze
Sponsors
Coming Soon: The CloudBees Newsletter for Jenkins
 Please complete the Jenkins survey to help us better serve the community
(bonus: a chance to win an Apple TV!)