Transcript Slide 1
Automated Capacity Planning and Support for Enterprise Applications Dharmesh Thakkar School of Computing, Queen’s University, Canada Performance Activities in The Waterfall SDLC Requirements Design Implementation Verification Verification: Performance test need to be conducted to verify that the required performance objectives are met Maintenance 2 Performance Activities in SDLC Requirements Design Implementation Verification Maintenance: Continuous monitoring, capacity planning and tuning need to be done to ensure that the system continues to meet the performance objectives Maintenance 3 Overview of Thesis Requirements Challenge: Theagoal of performance testingon is not Build capacity calculator based the to find Challenge: Load test and bugs, butperformance to establish benchmarks model stress test workloads need to Implementation be compared against the customer workloads Modify the workload or the deployment Verification Phase Performance Activities The capacity calculator stress helps customer Requirements scenario to put excessive on one or Performance Load Tests determine required hardware resources to Tests more system 3 9 1 Run the application for resources long periods under support the anticipated workload high workloads Design 3 Performance Model Performance Design Analyst 2 Stress Tests Customers engage the vendor’s support analyst to resolve the issues with system 9 Implementation Maintenance Phase Performance Activities 4 Deployment Challenge: The support analyst needs to troubleshoot the customerissues deployment Performance arise in customers’ production environment, which need troubleshooting Verification 8 4 5 Customer Customer Engagement Repository Performance Issues! 6 7 6 Execution Logs Customer Support Analyst Maintenance 4 Overview of Thesis: Our Contributions Requirements Requirements Design Design Implementation Implementation A framework for building performance model based capacity calculator Verification Verification Phase Phase Performance Performance Activities Activities Performance Performance Tests Tests 33 33 11 Performance Performance Analyst Analyst Performance Performance Model Model A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China. Acceptance rate: 26% Regular Regular Process Process Process Process Carrying Carrying Our Our Contribution Contribution Load Load Tests Tests 22 99 Stress Stress Tests Tests Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. Acceptance rate: 38% 99 Maintenance Maintenance Phase Phase Performance Performance Activities Activities 44 Deployment Deployment 44 88 55 Customer Customer Customer Customer Engagement Engagement Repository Repository Performance Performance Issues! Issues! 66 77 66 Execution Execution Logs Logs Customer Customer Support Support Analyst Analyst 5 Verification Phase: An Example of A Capacity Calculator 6 Concurrency 500 Creating Customer Profile Customer Login Search Title Purchase Title Creating Customer Profile Customer Login Search Title Purchase Customer LoginTitle T211 T111 T221 T231 T241 T121 T212 T213 T214 T112 T113 T222 T223 T224 T232 T233 T234 T242T122 T243 T123 T244 Creating Customer Profile Customer Login Search Title Search Title Purchase Title T311 T321 T131 T331 T341 T312 T313 T314 T322 T323 T324 T132 T133 T332 T333 T334 T342 T343 T344 Purchase Title Customer Profile Creating T141 T411 T421 T431 T441 T412T142 T413 T143 T414 T422 T423 T424 T432 T433 T434 T442 T443 T444 Concurrency 1500 Creating Customer Profile Concurrency 250 20 40 60 80 per hour) T111 Frequency T112 (transactions T113 T114 T121 T122 T123 T124 T131 T132 T133 T134 20 40 60 80 T141 T142 T143 T144 Concurrency 1000 Concurrency 250 Building a Measurement based Capacity Calculator Frequency (transactions per hour) Customer Login Search Title Purchase Title T114 T124 T134 T144 7 Performance Modeling Challenges ■ Large number of tests to be executed ■ Limited time for performance modeling ■ Manual process of setup, execution and analysis ■ Risk of error and repeating the full modeling process 8 Addressing the Challenges of Performance Modeling ■We created the framework to: –Automate the process –Speed up the process –Focus research on specific areas 9 Framework for Building Model based Capacity Calculator Test Enumeration Test Reduction Environment Setup Test Execution Test Transition Test Analysis Model Building Performance Model Performance Data Control Flow Data Flow ■ Case Studies: – Dell DVD Store – A large enterprise application 10 Customization Cost Test Enumeration Test Reduction Environment Setup Test Execution Test Transition Test Analysis Model Building Performance Model Minimal Minimal Minimal Minimal Minimal Minimal Minimal Another Build Extensive Reasonable Extensive Extensive Reasonable Reasonable Minimal Another Application 11 Maintenance Phase: Customer Support Customer Engagement Report Create Store Application Support Analyst Customer Engagement Repository Contains Symptoms Identified Problems Execution Logs Attempted Workarounds Solutions 12 Retrieving Relevant Reports ■ State of Practice: – No systematic techniques to retrieve and use information for future engagements – Keyword searching is limited: • depends on the search skills and experience of the analyst and peculiarity of the problem 13 Customer Support Problem Statement ■ We want to find customers with similar operational and problem profiles ■ We can reuse prior solutions and knowledge New Customer Engagement Compare Heavy Email, Light Web, Light calendar Light Email, Light Web, Light calendar Light Email, Heavy Web, Light calendar Heavy Email, Heavy Web, No calendar Light Email, Light Web, Heavy calendar Other Customers 14 Using Logs for Customer Support ■ Execution logs are readily available and contain – Operational Profile: usage patterns (heavy users of email from device, or to device, or light users of calendar, etc.) – Signature Profile: specific error line patterns (connection timeout, database limits, messages queued up, etc.) ■ Find the most similar profile 15 Execution Logs ■ Contain time-stamped sequence of events at runtime <time> Queuing new mail msgid=ABC threadid=XYZ <time> Instant message. Sending packet to client msgid=ABC threadid=XYZ <time> New meeting request msgid=ABC threadid=XYZ <time> Client established IMAP session emailid=ABC threadid=XYZ <time> Client disconnected. Cannot deliver msgid=ABC threadid=XYZ <time> New contact in address book emailid=ABC threadid=XYZ <time> User initiated appointment deletion emailid=ABC threadid=XYZ ■ Readily available representatives of both feature executions and problems 16 Example 4000 3000 C1 2000 1000 0 New Support Request 2500 2000 1500 1000 500 0 Compare 2500 2000 1500 1000 500 0 C2 4000 C3 3000 2000 1000 0 Other Customers 17 Technique to Retrieve Relevant Engagement Reports Customer Execution Logs Convert Log Lines to Events Event Distribution Identify Signature Events Signature Event Distribution Compare Event Distributions Closest Customer Engagement Reports wrt Operational Profile Customer Engagement Repository OUTPUT RESULT SET Compare Event Distributions Closest Customer Engagement Reports wrt Signature Profile 18 Case Studies ■ Case Study I – Dell DVD Store open source application – Code instrumentation done for event logging – Built the execution log repository by applying synthetic workloads, changing the workload parameters each time ■ Case Study II – Globally deployed commercial application – More than 500 unique execution events 19 Case Study Results ■ Dell DVD Store – 100% precision and recall on both operational profile based and signature profile based retrieval ■ Commercial Application – 100% precision and recall for signature profile based retrieval – Results for operational profile based retrieval: Experiment Single Feature Group Multiple Feature Groups All Feature Groups Real World Log Files All the Log Files K-L Distance Cosine Distance Count of Log Files Precision Recall Precision Recall 28 67.71% 90.28% 67.71% 90.28% 28 60.71% 80.95% 75.00% 100.00% 12 72.92% 97.22% 62.50% 83.33% 12 54.17% 72.22% 68.75% 91.67% 80 59.93% 79.90% 56.72% 75.62% 20 Sources of Errors ■ Events that do not correspond directly to a particular operational feature, such as idle time events, server health check events, startup and shutdown events ■ Imbalance in the event logging 21 Requirements Requirements Summary: Our Contributions Design Design Implementation Implementation Verification Verification Phase Phase Performance Performance Activities Activities Performance Performance Tests Tests 33 33 11 Performance Performance Analyst Analyst Performance Performance Model Model A technique for helping troubleshoot performance issues by retrieving relevant report from a customer engagement repository 22 Regular Regular Process Process Process Process Carrying Carrying Our Contributions Contribution Our 99 Presented at the Workshop on Software and Performance (WoSP 2008) Princeton, NJ. Stress Stress Tests Tests 99 Maintenance Maintenance Phase Phase Performance Performance Activities Activities 44 Deployment Deployment 44 88 55 Customer Customer Presented at the International Conference on Software Maintenance (ICSM 2008) Beijing, China. A framework for building performance model based capacity calculator Load Load Tests Tests Customer Customer Engagement Engagement Repository Repository Performance Performance Issues! Issues! 66 77 66 Execution Execution Logs Logs Customer Customer Support Support Analyst Analyst 23