Transcript Very cool title goes here and look you have plenty of room

Deploying Tomcat Clusters in an
Advanced Blackboard Environment
TM
Speaker: Steve Feldman
Director, Software Performance Engineering and Architecture
[email protected]
© Blackboard, Inc. All rights reserved.
Agenda and Introductions
»
»
»
»
»
»
Goals, Objectives and Outcomes
Background Knowledge
Hands on Learning
Benchmark Statistics
Questions…Questions…Questions
Total Time: 50 Minutes
2
Note to Attendees
»
Tomcat clustering is not a standard configuration of the
Blackboard Academic Suite.
»
»
»
Clusters have been set-up and tested by Blackboard Engineers,
but have not been certified as a valid configuration.
The technology is proven and used throughout the world.
The same can be said of server virtualization
»
»
Many Blackboard clients use and deploy virtual servers.
The guidance provided in this presentation is to help clients take
advantage of the power of clustering, but help clients understand
that they must have knowledge on setting up and managing
clusters.
3
Presentation Goals
»
The goals of this presentation are:
»
»
»
Introduce the concepts and techniques for planning and
deploying Tomcat Application Clusters.
Help administrators understand the value of Tomcat
clusters for performance and availability.
Show administrators how to set-up their very own
Tomcat cluster.
4
Presentation Objectives
»
»
»
»
»
»
»
Define what are clusters.
Define what are nodes.
Explain the differences between cluster nodes and load
balanced nodes.
Review the configuration requirements for setting up a
Tomcat cluster
Review the configuration requirements for setting up web
server distribution to the cluster.
Explain issues that might come up with clusters and
provide techniques for diagnosing and resolving the
issues.
Convince every customer to go home and start setting up
clusters in their development environments.
5
Presentation Outcomes
»
At the end of the session, administrators will be
able to do the following:
»
»
»
Define the difference between a load-balanced
configuration and a tomcat cluster.
Set-up a Tomcat cluster or a load-balanced nodes
Configure clustering for quality of service.
6
What is a Tomcat Cluster?
A cluster is a collection of two or more Tomcat
instances running in parallel and working together
to provide robust scalability, performance and
reliability.
» Clusters can be multi-homed and reside
physically on the same hardware.
» Clusters can be distributed and reside on
distributed hardware.
» Clusters are indistinguishable to end users
presenting a seamless experience
»
7
Why Cluster?
»
Clusters distribute workload
»
»
»
Clusters can help better utilize the hardware you run on
»
»
Clusters enable a scale-up model to go along with load-balanced
installations (scale-out model)
Clusters can provide additional availability
»
»
Minimize unnecessary queuing
When configured for a small JVM it can reduce the amount of JVM
pause times.
Not recommend using Tomcat Clusters as the primary failover
solution
Clusters are cool…and because they are cool you should
consider them!
»
Just one person’s opinion!
8
Visual Look at a Tomcat Cluster
Traffic is directed by the
Web Server like a software load balancer
Web Server
Instance
Node 1
Nodes know little
about there partners
Node N
Node 2
Recommend 1 node
Per Core
9
What is an Application Node?
An application node is a single instance of
Tomcat.
» Nodes can be part of a cluster.
»
Means they are aware of each other
» Part of a group
» Failover
»
»
Nodes can be part of a load-balanced
configuration.
»
A single physical server can have multiple logical nodes
not in a cluster, but technically load-balanced by JK
10
Clustered Nodes versus Load-Balanced Nodes
»
A load-balanced node is the following:
»
»
»
»
Physically distinct from a hardware perspective.
Has its own web server managing traffic.
Has a dedicated configuration and resource definition.
Has no understanding of failover definition.
»
»
Relies on third-party resources like load-balancers.
A clustered node is the following:
»
»
»
»
Can be physically or logically distinct from a hardware perspective.
Shares a web server with potential N+ nodes.
Shares configuration and resource definition with other nodes.
Has failover definition as part of the application configuration.
11
Load-Balanced and Clustered Nodes
Nodes aware of
each other.
Hardware or Software
Load Balancer
Cluster
Nodes unaware
of each other.
Load-Balanced
Web Server
Instance
Node 1
Web Server
Instance
Node N
Node 2
Node 1
Node N
Node 2
12
Session Replication
»
To enable session replication in Tomcat, three
different paths can be followed to achieve the
exact same thing:
»
»
»
»
Using session persistence, and saving the session to a
shared file system (PersistenceManager)
Using session persistence, and saving the session to a
shared database (JDBCManager)
Using in-memory-replication, using the
SimpleTcpCluster that ships with Tomcat 5
(server/lib/catalina-cluster.jar)
Recommend Using SimpleTcpCluster
13
Guidelines for Setting Up a Cluster
»
»
The cluster configuration is completely described in the
server.xml file under
$BLACKBOARD_HOME/apps/tomcat/conf
You can set-up a true cluster or settle for load-balanced
JVMs.
»
»
Either way you win.
Create multiple clusters in a multi-homed configuration
initially
»
Each multihomed cluster is capable of failing over to each other
rather then across physical systems
»
Eliminates the need for cross-over connections between servers.
14
Ingredients of a Tomcat Cluster
»
Class Name= Must have class package
»
»
»
»
»
»
org.apache.catalina.cluster.tcp.SimpleTcpCluster
Cluster Identification and Listen Port
Multicast Communication Address
JK Connector Port
JK Configuration Definition for Load Distribution
A Few Directory Copies
»
»
»
»
$BLACKBOARD_HOME/apps/tomcat/bin
$BLACKBOARD_HOME/apps/tomcat/conf
$BLACKBOARD_HOME/apps/tomcat/work
$BLACKBOARD_HOME/apps/tomcat/log
15
Example Cluster Code
<Cluster className="org.apache.catalina.cluster.tcp.SimpleTcpCluster"
managerClassName="org.apache.catalina.cluster.session.DeltaManager"
expireSessionsOnShutdown="false"
useDirtyFlag="true"
service.mcastBindAddress="127.0.0.1"
notifyListenersOnReplication="true">
<Membership className="org.apache.catalina.cluster.mcast.McastService"
mcastAddr="228.0.0.4"
mcastBindAddress="127.0.0.1"
mcastPort="45564"
mcastFrequency="500"
mcastDropTime="3000"/>
<Receiver className="org.apache.catalina.cluster.tcp.ReplicationListener"
tcpListenAddress="auto"
tcpListenPort="4000"
tcpSelectorTimeout="100"
tcpThreadCount="6"/>
<Sender className="org.apache.catalina.cluster.tcp.ReplicationTransmitter"
replicationMode="pooled"
ackTimeout="15000"/>
<Valve className="org.apache.catalina.cluster.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
</Cluster>
16
Easiest Way to Skin a Tomcat Cluster
»
Copy $BLACKBOARD_HOME/apps/tomcat to
$BLACKBOARD_HOME/apps/tomcat_node2
»
»
Change all references of $BLACKBOARD_HOME/apps/tomcat to now
use $BLACKBOARD_HOME/apps/tomcat_node2
Modify server.xml in new directory to reflect unique server
port for AJP13 connector and shutdown.
»
»
Cluster ports need to be unique in order to create a socket unless
IP Addresses are defined and uniquely defined.
Modify CATALINA_PID in blackboard-tomcat.sh or .bat file
to point to the right node and log file directory.
17
Setting Up JK Load Distribution
Apache and IIS will route traffic based on JK
definition.
» Best to set-up a new workers.properties
»
»
Located under
$BLACKBOARD_HOME/apps/tomcat/conf/jk/workers.properties
»
»
Change httpd.conf to point to new workers.properties
(Unix)
Managed from within IIS
»
»
Blackboard Web Site  Jakarta
Simply copy workers.properties and rename old file
18
Example Workers.Properties
# One virtual worker
worker.list=loadbalancer
# Configure the virtual worker as a load balancer
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tomcat1,tomcat2
worker.loadbalancer.method=Request
# Attempt to
worker.loadbalancer.sticky_session=1
# Actual workers, not available directly
worker.tomcat1.port=8009
worker.tomcat1.host=localhost
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=50
worker.tomcat2.port=8209
worker.tomcat2.host=localhost
worker.tomcat2.type=ajp13
worker.tomcat2.lbfactor=50
19
Web Server Configuration
»
Apache manages Tomcat worker information in
httpd.conf
»
»
»
»
Define workers.properties
Establish JK Mount (Bottom of File)
Probably called new worker list “loadbalancer”
Windows manages Tomcat worker information un
uriworkermap.properties
»
Must note new worker name for Cluster (aka: worker
list)
20
Things to Consider
»
Apache has a limit of 1024 maxclients
More then enough for a 2, 3 or 4 node cluster
» Must always keep maxclients = (Tomcat Max Server
Processes) * (# of Tomcat Nodes per Cluster)
» Recompiling Apache is one way to solve the problem
»
21
Advanced Feature: Setting Up QOS Models
Change
JKMount
Application Server
**JK Mount says /webapps/* but
can be changed to any single
webapp if you choose.
Apache / IIS
Tomcat
Modperl / PerlEx
Tomcat
JDBC
DBI
22
Example Benchmark Statistics
T2000 Analysis and Clustering
20,000
18,000
Sessions Per Hour
16,000
14,000
12,000
Single Server 2 Nodes
10,000
Load Balanced 1 Node Each
Load Balanced 2 Nodes Each
8,000
6,000
4,000
2,000
0
Configuration
23
Example Benchmark Statistics
Workload
R7.1 Entry-Level R7.1 Mid-Level
R7.1 High-Level
R7.1 HL
Clustered
R1 (Workload of 120 7238 Sessions/Hr 8080 Sessions/HR 8212 Sessions/HR 10455 Sessions/HR
Possible Concurrent
19 UPL/Second
22 UPL/Second
22 UPL/Second
25 UPL/Second
Simulations Learning
311,656
480,824
488,168
544,673
System/Community
Bytes/Second
Bytes/Second
Bytes/Second
Bytes/Second
System)
51,888 Transactions 53,780 Transactions 54,049 Transactions 59,239 Transactions
(2 Nodes)
R2 (Workload of 240 12459 Sessions/HR 13341 Sessions/HR 14913 Sessions/HR 16063 Sessions/HR
Possible Concurrent
31 UPL/Second
34 UPL/Second
33 UPL/Second
45 UPL/Second
Simulations Learning
640,958
729,616
695,319
968,128
System/Community
Bytes/Second
Bytes/Second
Bytes/Second
Bytes/Second
System)
87,433 Transactions 90,353 Transactions 94,181 Transactions
106,659
Transactions
(4-Nodes)
R3 (Workload of 360 17288 Sessions/HR 18455 Sessions/HR 20343 Sessions/HR 24034 Sessions/HR
Possible Concurrent
42 UPL/Second
50 UPL/Second
51 UPL/Second
65 UPL/Second
Simulations Learning
901,103
1,102,667
1,145,440
1,329,037
System/Community
Bytes/Second
Bytes/Second
Bytes/Second
Bytes/Second
System)
118,754 Transactions 130,811 Transactions145,287 Transactions
157,629
Transactions 24
(6-Nodes)
Example Benchmark Statistics
Workload
R7.1 Entry-Level
R7.1 Mid-Level
R7.1 High-Level
R7.1 HL
Clustered
R7 (Workload of 200 5721 Sessions/Hr 12548 Sessions/Hr 12974 Sessions/Hr 13804 Sessions/HR
Possible Concurrent
13 UPL/Second
33 UPL/Second
35 UPL/Second
36 UPL/Second
Simulations Full
275,672
728,082
735,846
763,955
Academic Suite)
Bytes/Second
Bytes/Second
Bytes/Second
Bytes/Second
37,313 Transactions 84,004 Transactions 84,970 Transactions 90,941 Transactions
(4 Nodes)
R8 (Workload of 400 11908 Sessions/Hr 18857 Sessions/Hr 14668 Sessions/Hr 24034 Sessions/Hr
Possible Concurrent
34 UPL/Second
53 UPL/Second
32 UPL/Second
65 UPL/Second
Simulations Full
668,189
1,157,486
676,802
1,392,037 Bytes/Sec
Academic Suite)
Bytes/Second
Bytes/Second
Bytes/Second
157,629
77,553 Transactions 118,353 Transactions 96,742 Transactions
Transactions
(2 Nodes)
(4 Nodes)
R9 (Workload of 600 12652 Sessions/Hr 23056 Sessions/Hr 20207 Sessions/Hr 27997 Sessions/Hr
Possible Concurrent
25 UPL/Second
63 UPL/Second
47 UPL/Second
71 UPL/Second
Simulations Full
451,975
1,196,553 Bytes/Sec
1,014,189
1,527,433 Bytes/Sec
Academic Suite)
Bytes/Second
Bytes/Sec
149,709 Transactions
181,121
64,289 Transactions
130,907
Transactions
Transactions
(6 Nodes)
25
(3 Nodes)
References
»
»
»
»
»
»
»
»
»
Tomcat: The Definitive Guide by Jason Brittain
http://www.onjava.com/pub/a/onjava/2004/01/28/tomcat5.html?CMP=I
LC-VP9761447835&ATT=804
http://tomcat.apache.org/tomcat-5.0-doc/balancer-howto.html
http://www.onjava.com/pub/a/onjava/2002/07/17/tomcluster.html?page
=1
http://www.onjava.com/pub/a/onjava/2004/03/31/clustering.html
http://www.onjava.com/pub/a/onjava/2004/04/14/clustering.html
http://arstechnica.com/articles/paedia/cpu/hyperthreading.ars
http://lists.samba.org/archive/samba-technical/2004December/038305.html
http://tomcat.apache.org/tomcat-5.0-doc/cluster-howto.html
26
Past Presentations of Note
»
B2 2006: How We Size the Academic Suite, Benchmarking at Blackboard
»
B2 2006: Deploying Tomcat Clusters in an Advanced Blackboard Environment
»
2006 BbWorld Presentation: Practical Guide to
Performance Tuning and Scaling (2 Hour Workshop)
»
B2 2005: Introduction to Load Testing, A Blackboard Primer
»
B2 2005: Performance Testing Building Blocks
»
Users Conference 2005: Managing Your Blackboard Deployment for Growth and
Performance
»
Users Conference 2005: Applied Software Performance Engineering
»
B2 2004: Introduction to Software Performance Engineering
»
B2 2004: Profiling Building Blocks for Performance Analysis
27
Questions?
28