Transcript Model based BES Capacity Calculator

A Framework for
Measurement based
Performance Modeling
Dharmesh Thakkar, Ahmed E. Hassan
School of Computing, Queen’s University, Canada
Gilbert Hamann, Parminder Flora
Enterprise Performance Engineering, Research In Motion (RIM), Canada
An Application of Performance Modeling
Concurrency
500
Creating Customer Profile
Customer Login
Search Title
Purchase Title
Creating Customer Profile
Customer Login
Search Title
Purchase
Customer
LoginTitle
T211
T111
T221
T231
T241
T121
T212
T213
T214
T112
T113
T222
T223
T224
T232
T233
T234
T242T122 T243 T123
T244
Creating Customer Profile
Customer Login
Search Title
Search Title
Purchase Title
T311
T321
T131
T331
T341
T312
T313
T314
T322
T323
T324
T132
T133
T332
T333
T334
T342
T343
T344
Purchase
Title Customer Profile
Creating
T141
T411
T421
T431
T441
T412T142 T413 T143
T414
T422
T423
T424
T432
T433
T434
T442
T443
T444
Concurrency
1500
Creating Customer Profile
Concurrency 250
20
40
60
80
per hour)
T111 Frequency
T112 (transactions
T113
T114
T121
T122
T123
T124
T131
T132
T133
T134
20
40
60
80
T141
T142
T143
T144
Concurrency
1000
Concurrency
250
Building a Measurement Performance
Model
Frequency (transactions per hour)
Customer Login
Search Title
Purchase Title
T114
T124
T134
T144
Performance Modeling
Challenges
■ Large number of tests to be executed
■ Limited time for performance
modeling
■ Manual process of setup, execution
and analysis
■ Risk of error and repeating the full
modeling process
Addressing the Challenges of
Performance Modeling
■We created a framework to:
–Automate the process
–Speed up the process
–Focus research on specific
areas
Framework for Measurement
based Performance Modeling
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Performance
Data
Control Flow
Data Flow
■ Case Studies:
– Dell DVD Store
– A large enterprise application
Model
Building
Performance
Model
Test Enumeratoin
Frequency (transactions per hour)
Concurrency
250
Search Title
Purchase Title
Concurrency
500
Creating Customer Profile
T211
Performance
Customer Login
T221
Data
Search Title
T231
Control Flow
Purchase Title
T241
T212
T213
T222
T223
T232
T233
Data Flow
T242
T243
T214
T224
T234
T244
Concurrency
1000
Customer Profile
Test Creating
Environment
Test
Customer
Login
Reduction
Setup
Execution
40
60
80
T112 Test T113 Model
T114
T122
T124
AnalysisT123 Building
T132
T133
T134
T142
T143
T144
Creating Customer Profile
Customer Login
Search Title
Purchase Title
T311
T321
T331
T341
T312
T322
T332
T342
T313
T323
T333
T343
T314
T324
T334
T344
Concurrency
1500
Test
Enumeration
20
T111
Test
T121
Transition
T131
T141
Creating Customer Profile
Customer Login
Search Title
Purchase Title
T411
T421
T431
T441
T412
T422
T432
T442
T413
T423
T433
T443
T414
T424
T434
T444
Performance
Model
Environment Setup and
Test Execution
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Performance
Data
Control Flow
Data Flow
■ Setup Environment and Test
■ Run Test
■ Shutdown Test
Model
Building
Performance
Model
Test Transition
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Performance
Data
Control Flow
■ Conduct tests
– Back to back
– Add a delay
– Use heuristics
Data Flow
Model
Building
Performance
Model
Test Analysis
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Model
Building
Performance
Model
Performance
Data
Control Flow
Data Flow
■ Validate the Test
– Check for bugs or run problems
■ Analyze Performance Counters
– Prune the warm-up and cool-down periods
Model Building
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Performance
Data
Control Flow
Data Flow
■ Create models using
– Regression techniques
– QN and other stochastic models
Model
Building
Performance
Model
Test Reduction
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Performance
Data
Control Flow
Data Flow
Model
Building
Performance
Model
Customization Cost
Test
Enumeration
Test
Reduction
Environment
Setup
Test
Execution
Test
Transition
Test
Analysis
Model
Building
Performance
Model
Minimal Minimal Minimal Minimal Minimal Minimal Minimal
Another
Build
Extensive Reasonable Extensive Extensive Reasonable Reasonable Minimal
Another
Application
Summary