Memory issues in production systems Production system Restricted
Download
Report
Transcript Memory issues in production systems Production system Restricted
Memory issues in
production systems
Production system
Restricted access
Application, DB, Application server, log files
Debugging, monitoring
Investigation should not affect users
Reproducing memory issue is not the best idea
Memory leaks
What is memory leak
java.lang.OutOfMemory: heap size
other symptoms - extensive garbage collection
Hard to discover
Hard to investigate
Users don't recognize it
Few information
Sometimes hard to fix
Design issues
Investigation of memory issues
Log files
Monitoring behavior of running application
application logs, access logs, gc logs
JConsole
Heap dump Analysis
JHAT, MAT, Netbeans
Log files
Motivation
find issues
find patterns leading to issues
Problems
combining different logs (access log, application
log)
bad visibility of root cause – not visible or hidden in
big amount of minor issues
memory problems trigger only few types of
messages
Investigating log files
Application logs
Access logs
Log free memory once per minute
Analysis of traffic and business use cases
GC logs
Show problems with GC (stop the world)
Monitoring via JMX
Tools
JConsole
Motivation
More information than in logs
Monitoring tests
Problems
Not always applicable on production system
(security reasons)
Test might not cover all scenarios
Heap dump analysis
Tools
Motivation
JMAP, JHAT, MAT, Netbeans
Offline investigation of behavior during specific time
frame
Comparison of different heap dumps
Problems
Access to heap dumps (security)
Creating Heap dump
jmap -dump:live,format=b,file=heap.dump <pid>
Part of JDK
Used to create heap dump on demand
JVM parameters
-XX:+HeapDumpOnOutOfMemoryError XX:HeapDumpPath=/directory/subdirectory
Produces file java_pid<pid>.hprof
Used for production system, creates heap dump when
OutOfMemory is thrown
Tools for heap dump analysis
JHAT
Part of JDK 6
Running as web application
IDE
Netbeans, MAT (Eclipse plugin)
Commercial tools
Heap dump analysis - JHAT
Heap Histogram
List of classes with number of instances and used
memory
Referrers vs Referees
Navigating in object graph
Example: Form → String → [c
Heap dump analysis – Netbeans,
MAT
Dominator tree
Shallow vs Retained heap
Object x dominates object y if every path in object
graph from root to y goes through x
Shallow heap is memory used by object
Retained set of X is set of objects which would be
removed by GC if X is removed by GC
Retained heap is sum of shallow heap in retained
set
Other queries
Leak suspects
Heap dump Analysis - OQL
Object query Language
Different dialect for each tool
JHAT
Simple SQL combined with javascript
callback functions
MAT
More SQL like (Subquery, union, distinct)
Predefined functions (dominator)
OQL Examples
JHAT
select
heap.findClass('org.apache.catalina.util.ServerInfo')
.statics.serverNumber.toString()
MAT
SELECT serverNumber.toString() FROM OBJECTS
org.apache.catalina.util.ServerInfo
Experiences
Bugs
Incorrect usage of external libraries or
frameworks
Design issues
Bugs
Map <String, String> redirectUrlCache = new LinkedHashMap<String, String>() {
@Override
protected boolean removeEldestEntry(Map.Entry eldest) {
return size() > 10000;
}
}
String getRedirectUrl(String url) {
if(redirectUrlCache.containsKey(url)) {
return redirectUrlCache.get(url);
}
String result = computeRedirectUrl(url);
redirectUrlCache.put(url, result);
return result;
}
Usage of frameworks
Hibernate
Short time usage of big amount of memory
Too big first level cache
Good understanding of framework is a must
Design Issues
Overused session scope
Big pages
Big buffer in page context
Page context referenced by tag handlers
Heavy load causes big memory usage
Fix might require changing use cases
Reusing functionality
Interesting links
http://blogs.sun.com/sundararajan/entry/querying_java_heap_with_oql
http://www.eclipse.org/mat/downloads.php
OQL help for JHAT
Standalone MAT
www.google.com