Transcript Usage Analysis Briefing

MITRE Performance
Testing: Load Testing
With Usage Analysis
The MITRE CI&T Performance Test Team
February 2009
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Acknowledgements

This briefing was prepared by adopting material from the
Fundamentals of LoadRunner 8.1 training manual

The final usage analysis method and model described in
this briefing was pioneered by the MITRE CI&T Performance
Test Team.

Initial methods and models were originated by Henry
Amistadi and evolved with the input of Chris Alexander,
Betty Kelley and Aimee Bechtle.
Page 2
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Purpose & Goal

Provide background on our team and load testing at
MITRE

Understand how the Project Lead can support load testing
throughout the performance testing process

Introduce our Usage Analysis methods so the Project Lead
1. Understands how we target a realistic load in load testing
2. Has confidence that the right amount of testing is being
performed on a project

Our goal is always to provide high quality, timely
performance tests that meet the needs of the customer
and the performance test team
Page 3
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Our Team

Comprised of 3 Performance Test Engineers and 1 Team Lead

Instantiated in 2001 as the Automated Test Team

Invested in Mercury (now HP):
– LoadRunner for Load Testing
– QuickTest Pro for Functional Testing
– Quality Center/TestDirector for requirements and test case
management

In the beginning customers suspicious of the reality and accuracy of our
testing

In 2005 stopped performing automated functional testing to focus on our
core competency, performance testing

Have matured from single application and environment testing to multiapplication and multi-environment testing

Preparing for a large-scale, enterprise load test for 2009-2010

Now, high level of confidence in our service
Page 4
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Background: What is Load?

What is Load? Traffic!!!!!
– Transactions distributed over time,
expressed as rates


Transaction Per hour (TPH)

Transactions Per minute (TPM)

Transactions Per second (TPS)
Why Load Test? To assess how well
your application is directing traffic
– Find the problems before your
customers do
– Prevent poor response times, instability
and unreliability (Road Rage!)
– The longer you wait to test the more
costly the problem may become
Page 5
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Background: When to Use Performance
Testing

When should I consider load testing? How can I use you throughout
the project?

Contact performance test team during the project planning process and
consider their incorporation into the plan
iSTEP Phase
Examples/Activities
Requirements
COTS Evaluation, Usage Analysis
Design
Prototyping, architecture and server evaluation
Implementation
Early assessment of application response times,
Support capacity planning
Test
Determining if the system is reliable enough for
production
Post Rollout
Isolating performance bottlenecks and degradation in
performance after production. Regression testing.

Load testing can be used throughout the product’s life
Page 6
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Background: The Process
Project Lead/
Developer
Performance
Expert
§
§
§
Funct’l
Expert
2
3
4
5
Plan Load
Test
Create
Scripts
Create
Scenarios
Execute
Tests
Analyze
Results
Desc Goals &
Objectives
Select Load
Test Types
Analyze
Application:
§
§
Define steps
Create
Scripts
Environment
Usage
Document
Plan
§
Supply Goals,
Obj, Schedule
Initiate Data
Collection
Select targets
Review Plan
§
Develop use
cases as input
into scripts
Provide Data
§
§
§
§
§
§
Supply
transaction
logs
§
Create
scenarios
based on
goals and
targets
§
Execute
scenarios:
Peak
Stress
Other
§
§
Target Transaction Rates
§
§
Sys and DB
Admins
1
§
§
Provide
Steps &
Sample Data
Review
scripts
§
§
Review
Scenario
design
Prioritize &
schedule tests
Tweak System
or Test
Based on
Analysis
Analyze
Data and
Test
pinpoint
Transaction
Rate
bottlenecks
Compare
Test
Transaction
Rate to
Target
§
Review
results
§
Change
application
§
Review
results
§
Change
settings (if
applicable)
Provide
Steps &
Sample Data
Review
scripts
§
§
Prepare Test
Environment
Monitor
resources
Page 7
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Plan Load
Test
Define Goals & Objectives

Begin working with performance test team upon completion of planning process

Start by discussing high-level goals & objectives.

And what are these objectives?
Objective
For Example…
Measure Application Response
Time
How long does it take to complete a business transaction?
Support Configuration Tuning
Which configuration provides the best performance level?
Regression
Does a new version of the software adversely affect
response time?
Assess Reliability
How stable is the system under a heavy work load?
Capacity Planning
Can the application support a growth in usage?
Identify Bottlenecks
What is the cause of degradation in performance?
Evaluate Products
What is the best product (COTS, hardware, etc.)?
Page 8
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Select Load Test Types
Plan Load
Test
What
are the types of load tests?
Type
Description
Typical or Peak Load
Determine whether system handles real world loads
Stress Load
Find the system’s breaking point. Test at 1.5, 2, 2.5, 3
times peak.
Exploratory or Goal
Meet predetermined criteria or “What ifs?”
Architecture
Verify architectural design, like failover, load balancing
Extreme/Volume Load
Check stability of system under extended periods of
high load
Page 9
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Plan Load
Test

Analyze System - Environment
Need to understand the logical and physical test environment to
make the proper tool, protocol selection and to know our constraints

What is the historical CPU utilization on these machines?

What other apps are on these machines?
Page 10
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Plan Load
Test

Analyze System – Usage Analysis
Usage analysis is appropriate when:
– Plan is to run a Typical, Peak or Stress test to meet the agreed
upon objectives
– There’s an existing system to collect information from

Usage Analysis is the process of calculating load targets from log
file data for the relevant URLS
– AKA Workload Analysis or Log File Analysis
– We employ a statistical evaluation of data collected

Results in a Transaction’s:
– Target Typical & Peak TPH
– Target Typical & Peak TPM
– Target Typical & Peak TPS
Page 11
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Plan Load
Test
Analyze System – Usage Analysis
A
Identify
Business
Trans.
§
Select key
business
processes to
be analyzed
and tested
B
Analyze
Coarse
Grain
§
§
Select the
days to be
analyzed
Collect log
files for
these days
§
§
§
C
D
E
Analyze
Fine Grain
Select
Targets
Convert TPS
Parse and
grep log file
data
Aggregate
data by hr,
min and sec
Create
cumulative
freq. distrib.
§
§
§
Review
cumulative
freq. distrib.
With PL
Select target
typical and
peak TPHs
and TPMs
§
Convert
TPS from
TPM using
Total Time
Distrib. or
Convert
TPS from
TPM using
Active Time
Distrib.
Page 12
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
A
Identify
Business
Trans.
Usage Analysis – Business Transactions

“A business transaction is a set of user steps performed within an
application to accomplish a business goal”

For example:
– Logon to Home Page
– Search
– Update profile

Focus on the business processes that are
– Most commonly used
– Business Critical
– Resource Intensive

We need the URLs that complete these business processes. These
are the target URLs

Also, what are your expected target response times for these
processes under typical load? Peak load?
Page 13
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
B
Analyze
Coarse
Grain
Usage Analysis – Coarse Grain

Purpose is to identify Typical Peak Usage
Patterns

Use WebTrends

We identify the Typical Peak by answering
questions such as:
– Is this application used cyclical or
quarterly?
– Are some months or days of the weeks
higher use than others?
– Is this application heavily used on a
daily basis?
– Is this application heavily used
throughout the day or only at specific
hours?
– What are the highest use days and
hours?
Page 14
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
B
Analyze
Coarse
Grain

Usage Analysis – Coarse Grain
Example
Corporate Portal:
MyMII (Sun Portal) Daily Requests for portal/dt
WEBTRENDS DATA
200000
– Looked at weekdays from Oct – Dec
2006
180000
160000
140000
120000
– Compared daily and hourly page
views in WebTrends for portal/dt and
amserver/UI/Login
100000
80000
60000
40000
20000
0
18-Sep-06 25-Sep-06
2-Oct-06
9-Oct-06
16-Oct-06 23-Oct-06 30-Oct-06
– Ranked days in terms of daily totals
6-Nov-06 13-Nov-06 20-Nov-06 27-Nov-06 4-Dec-06 11-Dec-06 18-Dec-06
Requests Per Day
– Excluded two busiest days because
of abnormal patterns
– Selected next 8 busiest days for
further analysis
Page 15
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
C
Analyze
Fine Grain
Usage Analysis – Fine Grain

Once the Typical Peak timeframe is determined, log files from that
period are collected for more detailed analysis.

Relevant fields are isolated for inclusion, such as:
– Requesting URL (as agreed upon with the project team),
Response Code, Date and Time request completed, Referring
URL, User information
Page 16
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
C
Analyze
Fine Grain
Usage Analysis – Fine Grain
Work Hours Cumulative Frequency
HTTP 200 (/portal/dt)


The data is aggregated by time
frame and frequency analysis
is performed. For an example
please see: Usage Frequency
Analysis Example
The transaction rates are
placed in a cumulative
frequency distribution
TPH
TPM
TPS
Mean (Ave)
4499
75
2
Median (50)
5103
81
2
60.0
5214
86
2
70.0
5381
91
2
80.0
5532
96
3
90.0
5778
104
3
95.0
5896
110
4
97.5
5988
118
5
99.0
6075
131
5
100.0
6075
309
26
Page 17
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
D
Select
Targets
Usage Analysis – Select Targets

Select your Typical and Peak targets

We recommend using the TPH and
TPM rates at the Median, 50th
Percentile, as the Typical Transaction
Rates



Work Hours Cumulative Frequency
HTTP 200 (/portal/dt)
TPH
TPM
TPS
Mean (Ave)
4499
75
2
Median (50)
5103
81
2
60.0
5214
86
2
70.0
5381
91
2
The TPS in this analysis is a reference
point.
80.0
5532
96
3
90.0
5778
104
3
This index is a satisfaction index:
95.0
5896
110
4
– What percentage of the time are you
willing to have users be
satisfied/unsatisfied?
97.5
5988
118
5
99.0
6075
131
5
– If you design a system to handle 131
TPMs your users would be satisfied with
response time performance 99% of the
time
100.0
6075
309
26
We recommend using the TPH and
TPM rates at the 99th percentile as
Peak Transaction Rates
Page 18
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
E
Convert TPS
Usage Analysis – Convert TPS

Purpose is to know the target TPS, and at what time intervals, the transactions
are going to occur in the test

We can choose from two methods
– Method 1: Total Time Distribution (How?)
Method 1 Total Tim e Distribution
2.5

Assumes transactions are evenly
distributed and every minute or second is
active
1.5
actual trans
avg trans
1
0.5
Represents the lowest transaction rate in
terms of load distribution
– Method 2: Active Time Distribution

2
TPS

0
9:00
9:01
9:02
9:03
9:04
9:05
Tim e Sam ple
(How?)
Method 2 Active Tim e Distribution
Accounts for the wave patterns of
transaction activity
2.5


Assumes that not every minute and
second is active
The active and inactive times are
calculated
TPS
2
1.5
actual trans
avg trans
1
0.5
0
9:00
9:01
9:02
9:03
9:04
9:05
Tim e Sam ple
Page 19
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
1
Plan Load
Test
Pulling It All Together

At the conclusion of the usage analysis we meet with the Project Lead and
review the plan.

Ideal if conducted during development, prior to integration testing

Review Business Processes and Target Transactions Rates

Target Transactions Rates Example: “The system shall sustain a load home
page transaction time of 3 seconds or less during typical hours at a rate of 2.02
transactions per second, and 3-5 seconds maximum during peak hours at a rate
of 3.3 transactions per second.”
Transaction
Typical Targets
TPH
Portal Home Page
5103
TPM TPS
PRT*
81 2.02 <3s
Peak Targets
TPH
TPM
6075
131
TPS
PRT*
3.3 3-5s
* PRT = Preferred Response Time
Page 20
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
2
Create
Scripts
Create Scripts

Scripts contain the recorded steps, test data, and think times that define
the transactions

Functional test cases, use cases, design documents and training
manuals are a good source for this information

A sample of the detail we need:
Step
Data
Click link for “My Insurance Plan Summary”
Username
Logon
Password
Click on the “Life” link under Type of Benefit
Click on the Edit button under Covered Beneficiaries Section
Enter New Primary allocation box for first beneficiary
100
Click “Save”
Click “Ok”

A real example: Detailed Steps
Page 21
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
3
Create
Scenarios
Create Scenarios

Scenarios define the overall load test execution. The instructions.

Elements of a scenario are:
•
Scripts
•
Vusers: Emulate the steps of real users. Steps are recorded into a
script.
•
Ramp Up Rate: The rate at which vusers are suddenly or gradually
introduced into the test
•
Think Times: Emulates how long the user pauses when:
•
•
Executing a transaction or business process. I.e. pauses
between clicks or keystrokes
•
Between transactions (iterations), between starting the script
over
Test Duration: The length the test will run. 1 hour, 4 hours, 8
hours. Varies by test type.
Page 22
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
4
Pulling it All Together
Execute
Tests
Script
Home Page
(Default Config)
Scenarios
Test Runs
Typical Day 1x
(Home Default Config)
Test Run
2007-03-04-A
Home Page
(Default Config)
Bursts
Peak Second Bursts +1x
(Home Default Config)
Favorites
Typical Day 1x
95% Home (Default)
+ 5% Edits
Test Run
2007-03-09-F
RSS
Exploratory
Edits
Test Run
2007-03-09-H
Center Portlet
Exploratory
Project Services
Project Services
Typical Day 1x
(Home Config)
Test Run
2007-03-12-L
HomePage
(Configured)
Typical Day 3x
(Home Config)
Test Run
2007-03-12-N
Test Run
2007-03-07-D
Test Run
2007-03-09-I
Page 23
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
4
Execute
Tests


Execute Tests
Prepare Test Environment
–
Account Setup
–
Data Setup
Plan for Multiple test runs
–
Debug
–
Baseline
–
Full execution

Tests will be run in priority order as we agreed upon

Execute tests as a team!!!
–
Systems and Database administrators should monitor the performance
of each tier of the application
–
Ideally a network administrator could monitor the network
Page 24
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
5
Analyze Results
Analyze
Results

At the end of the test run(s) an analysis file is produced that reports:
•
Running Vusers
•
Duration
-
Response Times By Transaction:
•
o
Average & Maximum
o
Percentiles
Test Transaction Rates
-
Test Total Transactions (Passed/Failed)
-
Test Transactions Per Second

Example analysis file: Sample LoadRunner Analysis File

The Target Transaction Rates from the Usage Analysis are compared to
the Test Transaction Rates. If they don’t reconcile then…
Page 25
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Tweak System
or Test
Based on
Analysis

Tweak The Load Test or The System
The test will need to be updated and/or rerun if:
1. The test goals have not been met then change the script or scenario
–
Test Transaction Rates are not similar to Target Transaction Rates
- For an hourly test the Total Transactions should be close to the
Target TPH
- The TPM and TPS rates should meet or exceed the TPM or TPS
from the usage analysis
- Increase vusers and decrease think time to increase load
2. Performance has not met expectations (i.e. poor response time or CPU
utilization)

–
Change Code
–
Change System Configuration
–
Change Architecture
This is an iterative process which takes time. This needs to be planned
for.
Page 26
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Finally…
Roll
out and repeat!!
– When the load test and performance goals are met you are good
to go
– Contact performance test team when application changes are
made that affect performance:

Significant GUI or Code changes

Changes in the architecture or environment

Performance degradation in production

Adding more users
Page 27
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
In Conclusion…

Partner with the peformance test team:
– Engage team from the beginning – during planning
– Have goals and objectives about what you want your
performance test to accomplish
– Provide:

Architecture diagrams

Transactions and steps to be scripted

Test data

Webtrends and log file data

Server permissions and DBA/Sysadmin support
– Select Targets
– Review the test plan, prioritize and schedule the tests
– Plan enough time for an iterative testing process
– Keep team informed of schedule or resource changes
Page 28
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Background

Detailed Calculation Examples
Page 29
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Method 1: Even Distribution - Example

Evenly divide the TPH and TPM transaction rate by 60
§TPM = TPH/60
§TPS = TPM/3600
%

TPH
TPM
TPS
For example:
§TPM = 6075/60
–TPS = 131/60
101.3
2.2 (We would use this one)
Page 30
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Method 2: Active Time Distribution
Calculate Active Time from log
as the percentage of time
requests are written to the log
Active Time
hour
mins / hour
secs / min
6
60.0
19.0
7
60.0
34.1
8
60.0
45.4
– Active Minutes per Hour
9
60.0
46.7
– Active Seconds per Minute
10
60.0
46.0
11
60.0
44.3
12
60.0
42.5
13
60.0
44.5
14
60.0
45.3
15
60.0
44.7
16
59.3
40.2
17
60.0
28.8
Average
59.9
40.1
We
calculate the Active Time
per hour of day in terms of:
For
example:
– At 9 o’clock every minute in
the hour had at least 1
transaction, but every second
did not.
– Only 46.7 seconds of the
minute were active
Page 31
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
4b
Method 2: Active Time Distribution Example
Evenly divide the TPH transaction rate by Average Active Time

– TPM = TPH/Active Minutes in a Hour
– TPS = TPM/Active Seconds in a Minute
%

TPH
TPM
TPS
For example:
§TPM = 6075/59.9
§TPS = 131/40.1
101.4
3.3
Page 32
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.
Method 2: Active Time Distribution –
Example (con’t)
Add
Inactive Time--beginning point for determining think times or pacing
– Percentage of time when no requests are completed or written to the logs or the
time between completed requests
– It includes the processing time of the second request
We
calculate the Inactive Time per hour of the day in terms of:
– Inactive Minutes per Hour
– Inactive Seconds per Minute
Inactive
Time needs to be sprinkled evenly between the Active Times to
avoid creating a compressed hour
10-second
Snapshot
120-second
Snapshot
Page 33
A National Resource Working in the Public Interest
©
Apr-20 The MITRE Corporation. All rights reserved.