Unicenter Service Desk (USD)

Preliminary Scalability Recommendations for r11 Revised January 11 2007

Objective

Primary objective: present sizing and location recommendations for the following Unicenter Service Desk components: Primary Server MDB Secondary Server Domserver/Webserver pairs (on primary/secondary) Web and Java Clients IIS Servers (for large clients) Network infrastructure (latency and bandwidth) © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

The Architecture

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Unicenter Service Desk r11 Architecture

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Components

USD Primary Server (with webdirector in all our tests) USD Secondary Server(s) (optional) Domserver/Webserver pair (on above servers) MDB – may be local or remote to Primary Server Java Client (optional – not preferred, will go away) Web Client © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Documented Minimum Recommendations

CPU: Minimum: single processor, 2 GHz or better Preferred: dual processor, 2 GHz or better RAM: Minimum: 2 GB Preferred: 4 GB Disk Space: 20GB minimum but allow for incremental growth to accommodate MDB growth Java Client: single processor, 1 GHz (or better) with minimum 1 GB RAM © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

What you need to know

Location of MDB – local or remote Number of boxes at each location Bandwidth from each location to central/managing site NW latency – USD can do lots of round trips Network and firewall port/direction restrictions Failover/Fault Tolerance requirements Monitor CPU and memory usage, especially on Domserver/Webserver pairs – add more resource when waiting on these resources – add another pair if available memory and CPU © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Other Factors

Optimization and tuning tips to enhance scalability Dedicated or shared machines?

Planning for future growth Best practice guidelines for filtering, monitoring and policy can reduce load Workflow © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Architecting/Tuning Resources

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Resources

CPU Memory I/O Subsystem Network (bandwidth and round trip latency) These are interrelated – one problem can mask others SQL can do too much I/O so you add memory SQL will cache disk to memory and end up with too much CPU use Real problem could be a missing index © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

CPU Consumption Remediation

Add CPU(s) Remove load (fewer Webserver/DOMServer pairs) Tune load – look at customization/configuration and adjust Is 10M contacts bad Is 10M assets when you only need 10k bad Is logging everything bad Is reporting against production DB bad © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

I/O Subsystem Remediation

Split service desk disk from other applications Split SQL log, tempdb, data, index Add arms (more units in stripe set, faster stripe set) Split reporting to a separate DB Look at what I/O you are doing – may be logging level, audit, too much data or data not being used, index maintenance, shared MD, wrong phase of the moon © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Memory Consumption Remediation

Add memory Reduce load Look at why memory is being used – could be missing index, reporting against an online transaction oriented database © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Network Consumption Remediation

Top 3 things: measure – measure – measure Add Bandwidth or reduce latency – but fix the one that matters Remove load Deploy secondary server in geo © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Management Servers

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Management Servers - Topics

Primary Server Secondary Server(s) Domserver/Webserver pairs IIS Servers © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Primary Server Guidelines

Dual CPU preferred with 2GB RAM minimum (4GB desirable) Monitor CPU and memory consumption/availability on ALL servers – this is the best indication of load/reserve capacity © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Secondary Server

Dual CPU preferred with 2GB RAM minimum (4GB desirable) One Domserver/Webserver pair per CPU – 1GB RAM per pair One Domserver/Webserver pair for each 300 simultaneous users (range is 200-400 based upon type of load but it is unusual to need a pair for 200 users unless all are analysts).

Monitor CPU and memory usage – remove a Domserver /Webserver pair or add more resource if nearly full – add more pairs if both CPU and memory are available.

Multiple secondary servers are required for large installations or those with resource constraints on one secondary server Secondary servers may be placed within geographic locales © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Secondary Server – Geographic Locales

Secondary servers may be placed within geographic locales There is less bandwidth consumed from primary to secondary than from secondary to web client Balance cost of secondary with improved performance benefit of a local secondary server Poor or congested bandwidth from primary to a locale can drive use of a local secondary – but you still need reliable and reasonably fast access from secondary to primary Growth in round trip latency is a driver for needing a secondary © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Secondary Server – dom/web server pairs

CPU utilization Our tests at target load were in range of 20-40% utilization We do not expect variation in response time below 60-80% If response time is good, no need to add CPUs even at 80% If CPU utilization is consistently low and memory available consider adding an additional domserver/webserver pair to improve response time if an improvement is needed Memory utilization Our tests had effectively no disk paging Your mileage may vary but we favor minimizing disk paging Add memory if constrained or if paging is observed © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Secondary Server

Recommended best practice is multiple machines or Partitions verses one Big Machine Additional separate IIS servers may be required to maximize thruput (10k user test used four IIS servers) – estimate is one IIS server (low end server) for each 2500 users.

No point in using large hardware for IIS servers Local secondary servers can reduce WAN bandwidth need to primary but will not “fix” poor or very slow connectivity © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

The MDB

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

What is it?

MDB – single database containing common tables and multiple product-specific tables previously in separate product dbs Stores all USD data As # rows increases, table size increases as well as disk space Db does not just need db location disk space but also work location space (e.g., for sorting, temp and transient files w/in db.) © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Single or Multiple?

Multiple MDBs can be used to accommodate organizational requirements or network considerations Distributed MDB – component dbs on dif computers (e.g., remote or local MDB) Should have one enterprise MDB serving as central db (provides complete view of enterprise state that other products can use) See Federated MDB presentation for more information on multiple MDBs © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

MDB Planning Guidelines

Use of Reiser file system is NOT recommended (not suitable for large dbs) Incr. computer reqs as necessary for enterprise MDB when MDB is integrating info for multiple CA prods MDB is business critical! s/b highly available MDB on a cluster can be a performance benefit MDB on 64 bit is also a significant performance benefit Cannot be installed on Windows Domain Controller © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

The Clients

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Client Options

Java Client: optional – if installed - admin functions removed Web Client: integrated, web-based user interface for all administrative functions – new for r11!

Web Client is the strategic direction for future use and where new features will be added © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Scalability Testing

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Testing Guidelines

Performance was tested based on the following: Network latency (between primary and secondary) Affect of DMZ and firewall requirements/placement Potential for bottleneck if multiple users attach docs Identify “breaking points” - point at which performance started to degrade, suggesting additional resources were required Description of tests follows © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Perfmon Data

Collected from Primary and Secondary Servers Identifies: System CPU utilization System memory utilization USD CPU utilization USD Memory utilization For MS SQL all basic SQL statistics For Ingres IMA statistics and reports © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Silk Performer Results

Saved from boxes running Silk Performer scripts Identifies: # failed and completed transactions Page response time (min., max., ave.) # users started and halted Total # errors © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario #1: Web Engine Performance

Objective: identify max. # concurrent users supported per web engine before performance degrades Configuration: 1 Primary Server w/ 1 DOM server and web engine on a Quad server DOM server Web Engine Web Engine Web Engine Primary Server © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 1: Goal

Test: Simulate 300 analysts logging in to create issues and 700 users logging in to create requests with concurrent connections peaking at 900 © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 2: Web Engine & DOM Server

Objective: Compare performance of single, dedicated web engine/DOM server pairing vs. multiple web engines Configuration: 300 users in single DOM server/three web engines (on one quad), and in a three paired DOM Server/web engine environment DOM server DOM server DOM server Web Engine Web Engine Web Engine Primary Server © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 2: Goal

Users log in, create issues/requests, log out and repeat # concurrent users gradually increase (1 user/8 seconds) to breaking point, then stay at this level.

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 3: Multiple Servers

Objective: compare breaking point of running on multiple small servers vs. one large server Configuration: 3 DOM server/web engine pairs on Quad Primary Server DS DS DS WE Secondary server WE Secondary server Primary Server © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

WE Secondary server

Test Scenario 3: Goal

300 users log in, create issues/requests, log out, repeat # concurrent users increase (1 user/8 seconds) to breaking point and stay at this level © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 4: Max. Concurrent Users

Objective: identify maximum # concurrent users the web engine can handle before errors occur or users are halted Configuration: quad server, start w/ single Primary server/DOM server/Web Engine and scale up DS DS DS DS DS DS WE WE WE WE WE WE Primary Server Secondary Server © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Secondary Server…

Test Scenario 4: Goal

Add more secondary servers (up to 4), DOM Servers and Web Engines (up to 5 each) and scale up to Best Practice Add Web Director, configure web engines to use SSL and rerun job © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 5: Network Impact on Performance

Objective: determine impact of network physical/logical definitions on performance Simulate scenario 4 and identify latency between primary and secondary servers.

© 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 6: DMZ and Firewall Performance Impact

Objective: determine how performance is affected by DMZ and firewall definitions Configuration: duplicate scenario 5 and restrict SLUMP port # to mimic implementation of USD across DMZ or firewall (i.e., Primary Server in intranet, Secondary Server in DMZ) © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario 7: MDB Location

Objective: to determine performance impact of locating the MDB locally vs. remote Configuration: duplicate configurations (with the exception of MDB location) and simulated user actions © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Test Scenario: Migration Impact

Objective: to determine average time required to migrate data from USPSD 6.0 to USD r11 Identify elapsed time for data migration © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Summary of Results – r11

Sustained load of 10,000 concurrent users for 8 hour period None of the partitions were taxed at any time System was very responsive Ticket save time ranged from 1.5-3.4 seconds CPU utilization ranged from 20% -40% Tickets created at a rate of 3.1 per second (180/minute) This is the equivalent of 94.6 million tickets per year for a 24x7x365 helpdesk!

Latent capacity estimated at 20-50% (12k-15k users) © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.

Summary of Results – USPSD 6

Sustained load of 8,000 concurrent users for 8 hour period None of the partitions were taxed at any time System was very responsive Ticket save time was roughly 1 second CPU utilization ranged from 10% -15% Tickets created at an average rate of under 3 per second Our hardware was not stressed at this load level Latent capacity estimated comparable to r11 © 2005 Computer Associates International, Inc. (CA). All trademarks, trade names, services marks and logos referenced herein belong to their respective companies.