Transcript European Conference 99

PeopleTools Application Server &
BEA Tuxedo Internals
[email protected]
Go-Faster Consultancy Ltd.
http://www.go-faster.co.uk
Who am I?
 Independent Consultant
 Performance Tuning
 PeopleSoft / Oracle
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
2
Rules of Engagement
 Interrupt me:
—
—
—
if you think I have got something wrong
if you have a question
if you can’t hear me
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
3
BEA Tuxedo
 Historical Overview
 Tuxedo Internal Architecture
 PeopleSoft 3-tier client implementation
 How does PeopleSoft Internet Architecture (PIA) fit in
 Configuration issues
 Monitoring
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
4
Brief Historical Background
 In the beginning
—
monolithic
Presentation
Business
Logic
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
Data
Access
(DBMS)
23 January 2002
5
Client Server
 Two-tier client-server
Presentation
Business
Logic
SQL
Raw Data
Data
Access
(DBMS)
LAN
Client
Server
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
6
3-Tier Client Server
 “1st Generation”
GUI
+
Application
Logic
Data
Access
(DBMS)
Request
Response
LAN
Client
Business
Logic
Server
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
7
3-Tier Client Server
 “2nd Generation” - Tuxedo
Presentation
Presentation
Logic
TUXEDO Clients
Business Logic
TUXEDO
Messaging
Service 1
Service 2
...
Service n
Data Management
SQL
DBMS 1
...
DBMS n
TUXEDO Servers
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
Database Servers
23 January 2002
8
Where did Tuxedo come from?
 Bell Labs - 1993
 Transactions for
 UniX
 Extended for
 Distributed
 Operations
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
9
What is Tuxedo?
 It is the knots on the ends of the string!
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
10
PeopleSoft 2-Tier Client
Presentation
Logic
Panel
Load
Panel
Save
Field
Change
etc...
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
11
PeopleSoft 3-Tier Client
Tuxedo Server
Tuxedo Client
Presentation
Logic
Panel
Load
Panel
Save
Field
Change
etc...
Server Side
Client Side
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
12
PeopleSoft 2-Tier Client
 Client Connects Directly to the Database
—
—
1 database connection per window
inactivity timeout
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
13
PeopleSoft 2-Tier Client
 Client Connects Directly to the Database
—
—
1 database connection per window
inactivity timeout
 What happens in 3-Tier?
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
14
PS/Tuxedo Internal Structure
 1st Application Server Process Started
—
—
Bulletin Board Liaison (BBL)
Reads PSTUXCFG configuration file
 Shared Memory Segment
—
Bulletin Board (BB)
BB
BBL
PSTUXCFG
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
DATABASE
23 January 2002
15
PS/Tuxedo Internal Structure
 Workstation Listener (WSL)
—
—
Listens for connection request from clients on a specified port
Client initially contacts the WSL
WSL
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
16
PS/Tuxedo Internal Structure
 Workstation Listener (WSL)
—
—
Client needs to know where to find the WSL
Machine and port specified in Configuration Manager
WSL
BB
BBL
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
17
PS/Tuxedo Internal Structure
 Workstation Handler (WSH)
—
—
—
—
—
WSL
handles traffic from client after connection
Client is assigned to a port on the WSH
WSH port range can be constrained to match a firewall
Additional WSH can be spawned if necessary
Compress Messages
WSH
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
18
PS/Tuxedo Internal Structure
WSL
AUTHQ
PSAUTH
APPQ
PSAPPSRV
QCKQ
PSQCKSRV
QRYQ
PSQRYSRV
APIQ
PSAPISRV
SAMQ
PSSAMSRV
WSH
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
19
How does it work?
WSHQ
WSL
AUTHQ
PSAUTH
APPQ
PSAPPSRV
QCKQ
PSQCKSRV
QRYQ
PSQRYSRV
APIQ
PSAPISRV
SAMQ
PSSAMSRV
WSH
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
20
How does PIA work?
 Java Client & PIA use Jolt
—
—
Java Class Library
Jolt Listener maps Java functions to Tuxedo Messages
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
21
Jolt Internal Structure
AUTHQ
PSAUTH
APPQ
PSAPPSRV
QCKQ
PSQCKSRV
QRYQ
PSQRYSRV
APIQ
PSAPISRV
SAMQ
PSSAMSRV
JSHQ
JSL
WSL
JSH
JREPSRV
WSH
JREPOSITORY
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
22
New structure for PIA (Apache Webserver)
http 1.1
address+cookie
(stateless)
Sessionid =
“xy1k…vm8010
Cookie
Java
JVM
Servlet Engine
Servlet Container
Servlet
thread
IS stateful
Servlet
Webserver
HTTPD
Disk
(static
objects)
Servlet
Thread
Servlet
Thread
Servlet
Thread
Client
Client
mod_jserv
Port 8010
JSL
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
JSH
JSH
JSH
Tuxedo
Application
Server
23 January 2002
23
What is the difference between the web servers?
 Apache
—
—
Separate Servlet Engine
Separate encryption module (Raven)
 Weblogic
—
Contains an integral servlet container
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
24
Progressive Connection Concentration
35000 users
 Typical picture
 35,000 users
 1000 concurrent connections
—
—
—
—
1000 users active in the last n minutes
where n is the Java + JSH timeout
therefore need 1000 threads
but maybe only 4 JVMs
1000 connections
4 JVMs
1000 Java Threads
100 JSHs
 100 JSH
—
assumes 10 clients per JSH
 10-20 PSAPPSRV processes
—
each PSAPPSRV connects to database
10-20 PSAPPSRVs
1 database
 1 database
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
25
Configuration
psappsrv.val
psadmin
(configuration
dialogue)
psappsrv.cfg
ubbgen
(PS Utility)
psappsrv.ubb
tmloadcf
(tuxedo
process)
PSTUXCFG
BBL
(domain
startup)
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
psappsrv.ubx
23 January 2002
26
Configuration Files
 psappsrv.cfg
—
contains configurable parameters
 psappsrv.val
—
validation file for configurable values
 psappsrv.ubx
—
configuration template for Tuxedo domain
 psappsrv.ubb
—
configuration file for Tuxedo domain
 PSTUXCFG
—
—
compiled configuration file
read by BBL at domain startup
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
27
psappsrv.ubx -v- psappsrv.ubb
 psappsrv.ubx & psappsrv.cfg
—
—
—
PeopleSoft file
make changes to these files
difficult to read
 psappsrv.ubb
—
—
—
—
—
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
Tuxedo file
never make changes to this file
easier to read
this file is compiled by tmloadcf to
generate domain configuration file
PSTUXCFG
variables fully resolved
23 January 2002
28
psappsrv.ubb
 Five (or Six) Sections
—
Resources
–
—
resource limits -> BB sizing
Machine
–
—
Resource limits. Environment
(Network
– Connection between Machines)
—
Groups
–
—
Servers
–
—
Logical Split (BASE, APPSRV, JREPGRP, JSLGRP)
Number of servers, Command line
Services
–
Timeout, Load, Priority
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
29
Application Server Configuration Issue
 Move quick PSAPPSRV services into a second server (PSQCKSRV)
(y/n)?
—
—
<=PT7.58 this question is misleading
– services are additionally advertised on PSQCKSRV as well as
PSAPPSRV
– so they are advertised on more than one queue
>=PT7.59 and PT8.1
– services are de-advertised from PSAPPSRV
– quick services not used in PIA, only IC* services
– so not a problem
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
30
PeopleTools <=7.58
 Quick Service
WSL
APPQ
PSAPPSRV
QCKQ
PSQCKSRV
WSH
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
31
Inside the Bulletin Board
 Without Load Balancing (default)
—
—
—
—
Table of servers process where each
service is advertised
Read and populated as a stack
WSL looks for first free server where
request can be enqueued
This can cause problems
PSAPPSRV
PSQCKSRV
PSQCKSRV
PSQCKSRV
PSAPPSRV
PSAPPSRV
PSAPPSRV
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
32
Load Balancing
 Load Balancing is about choosing the queue upon which to place the
message
—
by default all requests have the same load
50
50
50
APPQ
PSAPPSRV
50
50
50
50
50
QCKQ
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
50
50
PSQCKSRV
23 January 2002
33
Load Balancing Trace
 Add -r to application server command line
—
CLOPT="-r -e APPQ.stderr -A -- -C psappsrv.cfg -D GP81O81 -S
PSAPPSRV”
—
-r enables trace to file ‘stderr’ (not channel 2)
-e overrides the output filename
—
—
process trace file with Tuxedo utility ‘txrpt’
obtain average service time
—
make load proportional to average service time
—
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
34
Load Balancing Trace
—
Output from TXRPT
SVCNAME
18p-19p
Num/Avg
--------------- -------SqlRequest
16/0.10
MgrClear
5/0.04
RamList
1/0.43
SamGetParmsSvc 1/0.50
--------------- ------TOTALS
23/0.12
TOTALS
Num/Avg
------16/0.10
5/0.04
1/0.43
1/0.50
------23/0.12
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
35
Load Balancing
 load = execution time (seconds) x 100
78
386
135
APPQ
PSAPPSRV
599
50
6
30
3
63
3
QCKQ
7
PSQCKSRV
116
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
36
Multiple Queues on Busy System
 You can have many server processes on a single queue
 BEA recommend no more that 10 server processes per queue
 If you have queue contention it is likely to impede the WSH/JSH
processes enqueuing requests. You are unlikely to observe queuing
with PQ.
 So, if you want more than 10 PSAPPSRV processes you should use
two APPQ queues
 In, which case you should to enable load balancing and you must
calculate the load.
—
Less significant with PIA because services different.
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
37
Multiple Queues on Busy System
WSL
APPQ1
PSAPPSRV
APPQ2
PSAPPSRV
WSH
BB
BBL
DATABASE
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
38
Multiple Queues on Busy System
 psadmin will not correctly calculate
—
—
MAXACCESSERS
MAXSERVICES
 tmadmin->bbs
—
will report maximum number of services
– count of all services advertised on each server process
 and if you have multiple work station listeners
—
MAXWSCLIENTS
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
39
Service Priority
 Priority is about queue jumping
—
—
—
servers de-queue the request
9 out of 10 de-queued on the basis of priority
1 out 10 is taken from the front of the queue
80
50
50
50
QCKQ
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
50
50
PSQCKSRV
23 January 2002
40
Service Priority
 All PeopleSoft Services have same priority
—
specified in SERVICES section in psappsrv.ubb
therefore there is no queue jumping
—
no reason to believe that it would be advantageous
—
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
41
Tuning Tip
 Financials
—
—
—
—
—
On-line journal edit & post - remote call
Service initiates a Cobol process
Process can take minutes
RemoteCall services can block APPQ
Move RemoteCall service to a separate queue
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
42
Multi-Node Domain
 1 domain on many nodes
DBBL
WSL
WSH
BB
APPQ
PSAPPSRV
BBL
BRIDGE
TLISTEN
DATABASE
BRIDGE
TLISTEN
BBL
WSL
WSH
BB
APPQ
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
PSAPPSRV
23 January 2002
43
Multi-Node Domain
 Distinct Bulletin Board Liaison process
—
Master Node
 TLISTEN process
—
Listens for incoming requests
 BRIDGE
—
—
Sends & Receives messages between nodes
in a load balancing environment
– configurable bridge load add to remote queues
– limiting number of messages sent across the bridge
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
44
Multi-Node Domain
 DO ATTEMPT THIS WITHOUT BEA CONSULTANCY
 At UBS
—
—
—
—
PT 7.5x
susceptible to network errors
integrated into HP service guard
ultimately this was removed from production because of transient network
errors between the nodes causing the domain to ‘partition’
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
45
So what is going on?
 tmadmin
—
—
Tuxedo command line interface
also available within psadmin
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
46
tmadmin commands
 pq - printqueues
—
—
amount of work queued and waiting
not always accurate
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
47
tmadmin commands
 psr - printservers
—
reports activity of individual server processes
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
48
tmadmin commands
 psc - printservices
—
each service on each server
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
49
tmadmin commands
 Lots of other tmadmin commands
 documented in the Tuxedo documentation
 Use them to
—
—
—
dynamically reconfigure the domain
dynamically resize the domain
– start/shutdown extra servers
monitoring scripts
– tmadmin -r for read only mode
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
50
Java Administrative Applet
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
51
Conclusion
 So now you know
—
—
—
—
—
—
—
—
BEA Tuxedo/Jolt is a messaging protocol
How the messages are moved around
How PeopleSoft has used it
How the PIA has been constructed
Options for Tuning and Sizing
Load Balancing
Multi-Node domains
Monitoring Scripts
 BEA Documentation
—
—
is on your CD
http://e-docs.bea.com
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
52
More Information about PT8 Application Server
 PeopleSoft 2000 Conference: Americas
—
—
8006: PeopleTools 8.1 Application Server
8010: Inside the Application Server
– Buzz Voelker
– PeopleTools Application Server Team
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
53
Final Caveat
 PeopleSoft will not support any of the changes that I have described
in this presentation
 Everything described in this presentation has been done, in
production, at UBS
 If you want to use a multi-node domain you are strongly
recommended to arrange support/consultancy direct from BEA.
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
54
Questions?
BEA Tuxedo Internals
Go-Faster Consultancy Ltd.
23 January 2002
55
PeopleTools Application Server &
BEA Tuxedo Internals
[email protected]
Go-Faster Consultancy Ltd.
http://www.go-faster.co.uk