Transcript Performance Pitfalls  Paul Lockwood About the Presenter  Über-geek  First computer in 1981  First fulltime IT job in 1989  Clipper/Dbase IV,

Performance Pitfalls
 Paul Lockwood
About the Presenter
 Über-geek
 First computer in 1981
 First fulltime IT job in 1989
 Clipper/Dbase IV, PowerBuilder, Java, C#
 Contracted most of career
 Lived/ worked in four different countries
 Vast exposure to different approaches
 Stand on Shoulders of Giants
Causes are Well Known
 Memory Issues
 Mainly leaks, sometimes churn/ GC issues
 I/O
 Disk, network
 100% CPU
 Runaway threads
 Bad SQL, very poor DB Design
 Blocking lock around cache reads
 Knock on effect of spin-waits, waiting on I/O
Performance Basics
 Orders of Magnitude
 Services are thousands, even millions of times
slower
 Architecture can guarantee poor performance
 Typical Tuners think in milliseconds
 We Look for Big Mistakes
 Generally Ignore:





Value/reference boxing
Static method vs v-table lookup
Most marshaling
Hash lookups
Concatenating strings
Removing Guesswork
Summary
Warm Average
Magnitude
InProcess
680ns
1
Named Pipes Pooled (10K requests)
35us
102
Named Pipes (Connect, Disconnect)
237us
103
Web Service Local [REST result]
1ms [1.6ms]
104
Web Service LAN [REST result]
5.5ms [4ms]
104
Web Service Internet [REST result]
80ms [35ms]
105
MySQL Local Pooled
0.5ms
103
MySQL Local Non Pooled
6.2ms
104
6ms
104
2,700ms
107
MySQL LAN Pooled
MqSQL LAN Non Pooled
1ns is one CPU Cycle on a 1Ghz Processor
Architectural Mistakes
 Overuse of SOA, DI/IoC, Reflection etc
 ~100ms per SOA call is typical
 All are difficult to debug
 Layer and compile-in often the best choice
 Homegrown Frameworks
 Since 2000s these mostly fail
 Architect’s continued control prevents rescue
 Buzzword/Resume Driven Architectures
Demos #1
 JMeter to load test
 IDE to attach, pause and view all
threads
N-Tier: Popular but slow
Web Servers
Application Servers
User 1
User 2
Firewall/ Load Balancer
User Gazillion
Session State Server
The Cloud (Services)
DataBase Servers
Often with Two or
Three-way replication
Demos #2
 WinDbg – Analyzing a Production Dump
 Screen Captures from Real World Tuning
Personal Quick Wins
 Find/ Remove Memory Leaks

Static HashTable is classic example


Even saw a Logging Sink with contentions!
ReaderWriterLocks are easy fixes
 Remove .Net Exceptions used in Logic
 Remove unnecessary Locking





Short circuit SOA/ Web Service calls
SQL Tuning, even just adding Indexes!
Very chatty SQL calls -> pure Server or pure client
‘Real OOP’ Code -> Trivial SQL Code
Stop Log.Debug writing to Database
 Cache Lookup data (lazy load Singleton with IDispose)

Consider Implementing ‘Clear Cache Page’
More Difficult Fixes
 Complex sproc vs 10klocs of C#/Java code
 Automated Developer Tests essential with complexity
 SQL Optimization
 Hints, really understand data, rewrite sprocs..
 3-Tier Library total re-write
 Fought Architecture Team in a multinational
 Native Memory Leak
 Very hard to reproduce caching issue
 Advice from others totally misleading
 IISDiag confirmed the leak, then JMeter isolated
 Code Simplifications
 Hard coded logic -> Table driven methods
 XMLDocument parsing -> xsd.exe/ XmlBeans
Locating the Problem
 Existing Excuses


Only happens in live Environments


We have special hardware
Cannot reproduce traffic

Hardware is not good enough, need 64Gb etc
Problem Domain is insanely complex
 Test machine




Replicate live data if possible
CPU Horsepower is not really important
IDE useful to compile/ debug
Exclusive access to database, test hardware!!
 Taking Dumps (Not Recommended)



WinDbg, sos.dll, adplus, IISDiag (auto scripts) etc
Like CSI, it’s a snapshot in time
Total experts only – sole job function perhaps?
Final Slide
 Feedback is much appreciated
 DotNetWorkaholic.com
N-Tier/ Distributed Objects
- The Mysterious Allure of distributing objects
“Objects have been around for a while, and sometimes it seems
that ever since they were created, folks have wanted to
distribute them. However, distribution of objects, or indeed of
anything else, has a lot more pitfalls than many people realize,
especially when they’re under the influence of vendors’ cozy
brochures.”
Martin Fowler (2002)
 Distributing Objects:
 Increases Expense
 Increases Complexity
 Reduces Performance
Scalability





Look at Performance first

Scalability seems easier/ sexier


Use common sense; ask why not
Any ORM should scale: tune/cache the hot paths


Most apps are mostly reads, few writes. i.e. Cache for reads
Async audit/ analytics tracking to other servers

How many people have ten to fifteen web/app servers in
production?
Product X, Y or Z won’t Scale! Really?
DataBase writes are an ideal Bottleneck
Horizontal vs Vertical scaling
Major Enemies



Session State
Pessimization (Pessimistic Optimization)
Insane Architects :)