Transcript Application Instrumentation & Management

Application and Performance
Management Techniques for J2EE
“Management Built In” for the Adaptive
Enterprise
Scott L. Williams HP
[email protected]
Agenda
•
•
•
•
•
•
Challenges: Testing, Management, & Repair of Applications
Embracing Management Throughout The Application Lifecycle
Management Tools for Operations
Instrumenting Your Applications For The Operational Environment
Demonstration of Application Instrumentation Tools
Call to Action
Challenges: Testing, Management,
& Repair of Applications
The Ideal: Simple, Quick Resolution & Repair
Requirements
Gathering
Design &
Develop
2.
Use “root cause”
analysis results to
narrow down
which software
contains bug. Fix
bug.
Testing
Staging
3.
Deploy bug fix to staging. Execute
regression tests to verify integrity of bug fix.
4.
Release bug fix to
production.
Production
1.
Application is
“hanging” in
production. Use
OV to do “root
cause” analysis
and send results
to developers.
No Shared “Context”  Difficult For IT/Ops To Correct Problems
the application
Application
Development
IT/Operations
error
occurs
No way to help developers find error
within the application!
Examples of common pain points today
• Application integration (many technologies, techniques,
etc.)
• Poor communication (business+dev+operations)
• Deployment failures (insufficient information)
• Difficulty configuring application security (technology,
process)
• Maintenance (again, insufficient shared information)
Most time and money is dedicated to
maintenance in production
IT Today
Source: HP IT department
Innovation
10%
Consolidation/
Integration
30%
Operation
60%
Removing The Barriers To Success
Free up developer resources
– Today: developers often spend too much time supporting their
applications when those applications degrade or fail in production
– Goal: spend less time supporting applications and more time creating new
ones (greater alignment with new business needs)
Free up IT resources
– Today: Large numbers of operations staff; manual processes;
“archeology” of poorly documented applications
– Goal: Well documented apps; automated troubleshooting; better
management at all levels (infrastructure, apps, business processes)
Drive complexity out of the software lifecycle
– Today: Hidden assumptions, poor documentation, “silos”
– Goal: Meta-data (documentation); “management built in” to applications;
context shared throughout entire lifecycle (business-developmentoperations communicate);
Adaptive Enterprise design principles
Simplification
Simplify applications and systems to
reduce complexity and risk.
+
Standardization Standardize the way IT assets are used
with common components and processes.
• Business
+
Modularity
processes
Improve performance by managing
infrastructure components discretely or
collectively.
+
Integration
Applied
consistently
across:
Easily manage and modify the environment
through a uniform system of relationships.
• Applications
• Infrastructure
What is “Management for the Adaptive
Enterprise?”
• Do more with less
• Remove barriers to success
• Shift IT Investment from maintenance to
innovation
• Be ahead of the competition
• Be competitive or be outsourced
Moving from maintenance to innovation
Today
3 years
from now
10%
30%
• In large enterprises
up to 75 % are
personnel or
services costs
40%
20-25%
Consolidation
35-40%
Operation
60%
• Hardware, software
and infrastructure
only 25 %
Source: HP IT department
Innovation
Embracing Management
Throughout The Application
Lifecycle
the application
Application
Development
IT/Operations
error
occurs
No way to help developers find error
within the application!
Shared “Context”  IT/Ops Can Correct Problems
the application
Application
Development
CONTEXT:
instrumentation points,
performance metrics,
app meta data, etc.
IT/Operations
Use “context” to communicate problem
back to application developers
error
occurs
Start Here
track
track
monitor
control
monitor
track
monitor
control
control
Roles:Who Manages Apps in Production?
• Application “Owners” (business sponsors)
– Customer Management
– Renegotiate Contracts
– Measure/Monitor Compliance with SLAs
• IT/Operations (traditional, platform management)
– Ensure health of hardware, OS, some middleware
– 1st line of defense when problems occur
• Application Managers (newer concept)
– Ensure health of application containers and custom applications
– 2nd line of defense. Can often alter/control/repair without needing dev support.
• Software Developers
– Last line of defense.
– When all else fails, they use all runtime context to debug & repair problems
Market Observations & Trends – Gartner
• Best-of-breed is most popular strategy.
• Application development & support are most
often responsible for application management.
• Custom applications are most need of
management
• “Gartner believes …that responsibility for
application management increasingly resides
outside of the traditional IT operations
organizations”.
management
40
App support
30
LOB
% 20
10
outsourcer
0
60
Best of breed
Single vendor
Outsource
Other
0
Approa ch
other
Group
30
25
80
20
IT operations
M ost m a na gem ent need
Preferred Technology Approa ch
% 40
App
development
Responsibility for app
Custom
20
% 15
10
5
0
E-m a il
ERP
J2 EE
CRM
other
Ty pe
Gartner “Application Management Poll Reveals New Trends” April 15, 2004
Whom HP OpenView Application
Management Targets Today
CIO
LOB Owner
Biz/IT
Liaison
LOB
Finance
Bus Proc
Owner
BSM
App Dev
App
Manager
App QA
IT Controller
AM
App Support
IT Ops
Manager
Network
Mgr
Service
Desk
Mgr
Infra Svcs
Manager
Server
Mgr
Software Developer
Tools:
“Enterprise
debugger”
IDE
Architecture Models
Application Owners
(business sponsors)
enterprise
business
“coredump”:
runtime: SLO’s,
call stack traces
KPI’s, etc.
performanceBusiness,
info
business
audit trail
processes
Tools:
“BPM/BAM Dashboards”
CRM/Service Desk
Application &
application Environment
Context
runtime:
JMX, WMI
transaction
traces,
etc.
operational
runtime:
mgmt events, infra.
“health”, etc.
Operations/IT
Application Management
Tools:
Application Troubleshooting (JMX info, etc.)
Service Management Tools
Tools:
Performance Monitoring
Operational Status Monitoring
Infrastru. Troubleshoot Tools
Management Tools for Operations
3 Pillars of Management
• Tracking
– observe aspects of a single thread across multiple components.
– technologies: logging & auditing
• Monitoring
– capture runtime and historical events of a particular component,
for the purpose of reporting and notifications.
– technologies: Real-time statistics (JMX), logging, & auditing
• Control
– alter the behavior of the managed component directly without
recourse to interrupt the operation of the component.
– technologies: JMX, custom techniques
What is “Tracking?”
•
•
•
•
•
•
Measurement of “operational performance”
Combination of JVM profiling & application logging
Often is “durable” recording of what occurred.
Like “frames in a movie”
Some tools could “play back” the recording.
Helpful during testing & staging
What is “Monitoring?”
• Measurement of “business performance” (SLAs, KPIs)
• Measurement of current “operational health” and
“business health”
• Much “context” (operational & bus) only known by
applications: so use application logging & JMX
• Some is “durable” recording of what occurred.
• Some is transient or “current snapshot”
• Management Events & Corrective Triggers
What is “Control?”
• Automated changes or repair of application.
• Manual changes or repair to application.
• Often in response to management events “fired” during
monitoring.
• Sometimes modifies operational aspects of applications.
• Sometimes modifies business aspects of applications.
• Integral to building an adaptive enterprise and adaptive
applications.
• J2EE Applications often expose JMX MBeans for this
purpose.
Management Products for “Tracking”
• HP OpenView Transaction Analyzer
• HP OpenView Internet Services – measures availability (and
response-time) by creating synthetic transactions (aka “robots”)
•
•
•
•
Borland ServerTrace
Actional (web services)
JVM-level Profiling Tools
Etc.
Management Products for “Monitoring”
• HP OpenView for Operations, Reporting, Graphing
• HP Service Desk
• HP OpenView Internet Services + OVTA (real end user
response times and volume metrics, essential for SLA monitoring -e.g. response-time < 5 secs AND volume < 10,000 TPS)
•
•
•
•
•
BMC
CA Unicenter
Microsoft Operations Manager & Performance Monitor
Actional (web services)
Etc.
Management Products for “Control”
•
•
•
•
•
HP OpenView for Operations
BMC
CA Unicenter
Microsoft Operations Manager
Etc.
Instrumenting Your Applications
For The Operational Environment
What should you manage?
• Think in terms of an application’s “management interface”
• Use SLAs, business requirements, & IT/operations needs to help you
decide what management to support
• Management is a combination of what can be done at the platform
(JVM, J2EE Containers, etc.) & what you need to code
• You only need to code context that isn’t available from the platform:
(business-specific context, operational statistics not gathered by
platform, etc.)
business management
Marketing,
Analysts,
Architects
Customers
contracts with
end users, service
level agreements,
etc.
operational management
Solution Architects,
Industry Trends, etc.
starting-point
management
“best practices”
operational management
new requirements
learned over time
through experience
IT
Operations
management
requirements
management tools
determine “dials” and “knobs” for mgmt
IT
Operations
manage…
are specifications for
Management Interface of Solution
management
requirements
communicate…
are specifications for
functional &
systems
requirements
Solution (business logic, data, etc.)
Management
Solution(s)
“Call Center” App
Management
Interfaces
IT
“HR” App
Management
Interfaces
Tracking
What To Do:
– Logs and trace for method “entry” and “exit.”
– Performance metrics (at JVM-level as well as app container level)
– These are like frame-by-frame recordings of program executions.
They are durable
Available Technologies & APIs:
– JDK 1.4 or later: use standard java logging
– JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )
– JVMTI (JVM Tooling Interface) for tracking/monitoring tools
Monitoring
What To Do:
– Logs and trace for operational as well as “business” statistics.
– Business audit logs for historical business data (store exchanged
business documents, business events, etc.)
– JMX and WMI for application instrumentation
– Performance metrics (at JVM-level as well as app container level)
Available Technologies & APIs:
– JDK 1.4 or later: use standard java logging
– JDK 1.3: use log4j or lumberjack ( java 1.4 logging backport )
– JMX (MBeans) – “custom” MBeans as well as container-provided
Control
What To Do:
– react to predictive “triggers” and actually change running servers
& code
– modify the behavior of servers & code “in flight” without shutdown
– solve application problems/issues without always requiring
developers
Available Technologies & APIs:
– Custom-coded “control point” management objects using JMX
(MBeans)
– Custom-coded “control points” using: config files, web services,
RMI, etc.
– Container-level “control points” accessible via JMX, config files,
etc.
– React to change of state of “control point” objects
Best Practices for coding mgmt interfaces
• Use JMX for operational and business “health” & as “control
points”
• Use JMX to provide “rollups” such as:
– averages over time
– business-specific stats: # of “high value” customers this
hour
– current operational load on system
– current business activity rollup: $ processing right now,
etc.
• MBeans are for monitoring & “control”: Change their state to
control the application.
Averages & Roll-Ups Belong Outside
of application’s source
• Could provide averages over time: “transactions per hour”, “high
value customers served today”.
• But!, better to put these calculated metrics outside of the
application and outside of MBeans:
– Management system configurations (OVO templates)
– Management “rules” engines, event engines, etc.
• Provide simple, atomic data through JMX. Aggregate it externally.
Don’t count on “native” JMX
Management Events
• With JMX you can subscribe to and generate events.
• JMX events are “seen” within the JVM. But aren’t sent
outside to the management system.
• Different management products have their own event
mechanisms. And WSDM (Web Services Distributed
Management) standard has another.
• If you will be in a heterogeneous environment with many
technologies, don’t use JMX or WBEM notifications.
• Consider not using “native” JVM management events
A Very Simple! Standard MBean
public interface MoviePreviewStatsMBean {
public int getPreviewDownloadsCount ();
}
public class MoviePreviewStats implements MoviePreviewStatsMBean, java.io.Serializable {
public MoviePreviewStats () {}
public int getPreviewDownloadsCount () {
return m_nPreviewDownloadsCount;
}
private int m_nPreviewDownloadsCount = 0
protected void resetPreviewDownloadsCount () {
m_nPreviewDownloadsCount = 0;
}
protected void incrementPreviewDownloadsCount () {
m_nPreviewDownloadsCount++;
}
}
Different “Flavors” of MBeans
Standard MBeans
– Useful when the structure of managed data is well defined and unlikely to change
– Named with “MBean” added on the interface
– Attributes with getter/setter methods
– When to use: When first using MBeans; when the management interface isn’t changing much.
Dynamic MBeans
– must implement “DynamicMBean” interface
– is like DII in CORBA – your MBean doesn’t need a method signatures that match how the outside
world “sees” it
– You supply the “MBeanInfo” meta data for the bean
– When to use: When you want a level of indirection between the signature of your MBean and the
actual implementation; When impl side is dynamic (at runtime as well as design often changing).
Model MBeans
– (A subtype of dynamic MBeans)
– “RequiredModelMBean” always available and implements the “ModelMBean” interface
– The infrastructure implements most of the state caching, methods, etc. you would normally write
yourself.
– When to use: when you want to configure MBeans from outside of the source code (like from XML
file)
Publishing Your MBeans
• If you want the outside world (management systems) to see your
MBeans, you “publish” them to an MBeanServer.
• Each MBean that is published is like a “pointer” to an actual Java
object.
• Consumers of MBeans never “touch” the actual MBeans. Instead they
ask the MBeanServer to talk to the MBeans.
• Some Best Practices:
– As often as possible, use the standard JMX API for this (for instance, BEA
lets you use an MBean EJB)
– Use unique “domain” names for each application.
Code Sample for Publishing an MBean
/**
* Create the MBean.
Our application will be responsible for
* updating its value throughout the running of the application. */
Employee employeeMBeanRef = new Employee ();
/**
* Access the MBeanServer. */
MBeanServer mbeanServer = MBeanServerFactory.createMBeanServer();
/**
* Create “object name” for Bean. Publish it into the MBeanServer, using
* "HR_domain" as the domain part of the MBean's name attibute. */
ObjectName beanName =
new ObjectName("HR_domain:Name=Employee&type=manager");
mbeanServer.registerMBean( employeeMBeanRef , beanName);
Demonstration of Application
Instrumentation Tools
JMX Metric Builder
edit configuration file
SPI Configuration:
user defined metrics
your application
HP OpenView
for Operations
JMX SPI
MBeans
(the management
interface)
Container
MBeans
J2EE Container
Call To Action
• Think about app instrumentation “holistically” rather than as
some APIs. (Think about the “management requirements.”)
• Work within a development lifecycle that embraces
“manageability built in” as fundamental to success
• Determine your management requirements and management
interfaces up front.
• Build your applications to support: tracking, monitoring, & control.
• Use technologies such as JMX, web services & SOA to help you
create flexible & robust applications for the adaptive enterprise.
Questions