Transcript Microsoft BizTalk Server End-to-End Performance Testing Guidance Ewan Fairweather & Paolo Salvatori BizTalk Customer Advisory Team Microsoft.

Microsoft BizTalk Server
End-to-End Performance
Testing Guidance
Ewan Fairweather & Paolo Salvatori
BizTalk Customer Advisory Team
Microsoft
Session Objectives
•
•
•
Describe how we run BizTalk
performance assessments
Demonstrate tools and techniques that can
be used during a BizTalk performance
assessment
Introduce the BizTalk Server Performance
Optimization Guide
Session Agenda
•
•
•
Assessing application performance
Key learnings
BizTalk Server performance assessment
− Methodology
− Tools
Tuning the Microsoft Platform
Using LogParser To
Generate KPIs For Your
Results
Example results spreadsheet from
a recent performance lab
Session Agenda
− Assessing application performance
− Key learnings
• BizTalk Server performance assessment
− Methodology
− Tools
Key Learnings
Platform Tuning
•
•
•
•
•
Turn off hyperthreading
Adjust Windows Server 2003 for best
performance
Use a separate disk for PAGEFILE
Stop unnecessary services (for instance,
Print Spooler)
Defrag volumes hosting BizTalk dbs
Key Learnings
Platform Tuning
•
•
•
Disable the SynAttackProtect feature
Prevent antivirus scanning
Networking
− Adjust TCP/IP parameters in the registry
− Use 1-gigabit networks
− Separate subnets for Mgmt, Data, and
Messaging
− Tune NICs for performance
− Memory for packet buffering
− Offload processing to NIC if possible
Key Learnings
Platform Tuning – CLR Hosting Threads
• To prevent thread starvation, BizTalk needs
•
enough threads allocated to the .NET thread pool
Controlled by per host reg key
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Servi
ces\BTSSvc$hostname\CLR Hosting
DWORD entry
Default Value
Recommended Value
MaxIOThreads
20
100
MaxWorkerThreads
25
100
MinIOThreads
1
25
MinWorkerThreads
1
25
Key Learnings
BizTalk Host Configuration
• Split up the sending, receiving, processing, and
•
•
tracking into multiple hosts
This technique allows each host to:
− Have its own set of resources such as memory,
handles, .NET thread pool
− Decreases the contention on the MessageBox Host
Queue tables
Avoid running too many host instances on the
same box
Key Learnings
SQL Performance
• Isolate BizTalk data and transaction log files
• Pre-allocate space to prevent autogrowth at run
•
•
•
•
•
•
time
Create multiple data files of the same size, one for
each available core on the SQL machine, for the
TempDb
Add filegroups and files to the BizTalkMsgBoxDb
Scale up the SQL Server resources
− CPU, disk, memory
Separate BizTalk Databases across servers
Multiple MessageBox databases
Configure DTA Purge and Archive job
Key Learnings
Low Latency
• Internal message queue
• MaxReceiveInterval parameter
•
•
•
− 500 ms default
BatchSize parameter
Turn off global tracking
Use inline sends
Key Learnings
Orchestrations (1 of 2)
•
•
•
•
•
•
•
•
Eliminate unnecessary persistence points
Minimize the internal state/size
Use the Call Orchestration shape for performance
Asynchronous mechanisms give flexibility but require a
MessageBox round trip
− Direct Bound Ports, Self-Correlating Ports and Start Orchestration
shape
Eliminate logical ports and use inline sends
Prefer distinguished field over promoted properties
Turn off unnecessary tracking:
− Message send and receive tracking
− Begin and end tracking shape
Define a performance counter for each orchestration type
Key Learnings
Orchestrations (2 of 2)
•
•
Avoid using XmlDocument for processing large
messages
− The memory amount required to load an XMLDocument is
up to 10 times the actual size of the message
Instead use one the following techniques:
// Use an XmlReader object to read the content of the message part
using (XmlReader reader =(XmlReader)xlangMessage[0].RetrieveAs(typeof(XmlReader)))
{
...
}
// Retrieve the content of the message part as a string
using (string message =(XmlReader)xlangMessage[0].RetrieveAs(typeof(string)))
{
...
}
// Deserialize the content of the message part to an instance of a custom .NET class
using (RequestClass request =(XmlReader)xlangMessage[0].RetrieveAs(typeof(RequestClass)))
{
...
}
Key Learnings
Pipelines (1 of 2)
•
•
•
Use PassThruReceive and PassThruTransmit
pipelines whenever possible
Avoid loading the entire document into memory using
an XmlDocument object
− Use a streaming approach instead
If you need to read the value of specific nodes inside
an XML message part, do not use the
SelectSingleNode method exposed by the
XmlDocument class
− Use an instance of the XPathReader class contained in the
Microsoft.BizTalk.XPathReader.dll assembly
− Use an instance of XPathMutatorStream class contained in
the Microsoft.BizTalk.Streaming.dll if you need to read and
eventually change the value of one or more elements.
Key Learnings
Pipelines (2 of 2)
•
•
•
•
•
•
Use an instance of the VirtualStream class
(Microsoft.BizTalk.Streaming.dll) when reading a large
message
Promote items to the message context only if you need
them for:
− Message Routing (Orchestrations, Send Ports)
− Demotion of message context properties (Send Ports)
“Write” to the context instead of “Promote”
Use the IPipelineContext.ResourceTracker to track and
dispose non-CLR resources
Instrument your source code to make your components
simple to debug
Avoid ordered delivery and transaction support
Optimizing BizTalk
Memory Usage
Streaming techniques that can be used
in pipelines
WCF Adapters
•
Whenever possible, use one of the following WCF-* Adapters.
−
−
WCF-NetTcp Adapter
WCF-Custom Adapter + CustomBinding with the binaryMessageEncoder
and tcpTransport Binding Elements.
Send Side (BizTalk Send Port):
Parameter
tcpTransport->ConnectionPoolSettings->maxOutboundConnectionsPerEndpoint
tcpTransport->maxPendingAccepts
tcpTransport->maxPendingConnections
tcpTransport->listenBacklog
Default
1
1
10
10
Suggested
try changing to 20
try changing to 20
try changing to 20
try changing to 20
Receive Side (BizTalk Receive Location):
Parameter
tcpTransport->maxPendingAccepts
tcpTransport->maxPendingConnections
tcpTransport->listenBacklog
•
Default
1
10
10
Suggested
try changing to 20
try changing to 200
try changing to 20
Add the serviceThrottling service behavior to a WCF-Custom Receive
Location and use the following settings:
Parameter
serviceThrottling->maxConcurrentCalls
serviceThrottling->maxConcurrentSessions
serviceThrottling->maxConcurrentInstances
Default
16
10
-
Suggested
try changing to 20
try changing to 20
leave the default
WCF Adapter
Optimizing WCF for performance
Session Agenda
− Assessing application performance
− Key learnings
• BizTalk Server performance assessment
− Methodology
− Tools
BizTalk Server
Performance
Optimization Guide
MSDN
http://msdn.microsoft.com/en-us/library/cc558617.aspx
Assessing Application Performance
•
•
The usual story
− We all know testing is important for every
solution
− Testing is neglected
This is a problem
− Solutions using BizTalk tend to be business
critical and tolerate little downtime
− Poor testing increases the risk profile of your
project
− This runs the risk of huge technical and political
problems
Minimize Project Risk
•
•
•
•
•
•
•
Unit testing
Effective testing
Functional testing
reduces your
Integration testing
project risk
User acceptance testing
Disaster recovery testing/dry runs
Overload testing
Performance assessment
Session Agenda
− Assessing application performance
− Key learnings
• BizTalk Server performance assessment
− Methodology
− Tools
BizTalk Performance Assessment
•
•
•
•
An end to end process with a set of goals
and objectives
Occurs over a set period of time
It requires a team of individuals with the
right skill sets to “tune the stack”
Iterative knowledge transfer process
Design & Deliver
POC Performance
Assessment
Operate
Pre-Live
Performance
Assessment
Optimize
Post-Live
Performance
Assessment
Scope
Engagement Summary
• Determine resource requirements
•
•
•
−
−
−
−
−
What scenarios are you testing
What technologies are involved
What adapters are needed
SAN requirements
32 bit vs. 64 bit
Other requirements
− Throughput
− Latency
Document all information!
− Guidelines, Goals, Constraints, etc.
Staff the right people
Scope
Defining Goals
• Provide objectives and success criteria for the lab
• Goals should cover
− Performance metrics
•
•
•
−
Throughput, Latency
Roles and Responsibilities
Deliverables
Define any constraints
− Is hardware fixed?
− Scale up/out BizTalk/SQL
− Can third party systems be scaled out?
−
Web servers, MQ Series, Custom DB’s
Scope
Defining Goals (Example)
Objective
Measure the Maximum Sustainable Throughput (MST) for the given
use cases with different hardware configurations.
Test Case 1
Messaging-Only scenario
Throughput : 50 messages/sec sustainable
Latency: response time < 2 sec
for 90% of messages
Test Case 2
Payment service BizTalk orchestration
Throughput : 3.6 million messages within
an 8 hour period (125 messages/second)
Latency : N/A
Scope
Logical and Physical Architecture
Plan
Determining Resource Requirements
Timeline
− 2 weeks of preparation time
− 1 week per test case
− 1 week to document findings
SMEs required
− MQSeries, SQL Server,
IIS, SAN Specialist
Hardware request
− SAN Storage (HBA, Luns), Networking, Clustering
Lab space
− Appropriate space, whiteboard, projector, security of room
Plan
Sample Timeline
Prepare
•
•
•
Detailed Solution Design
Code complete
Invest in test assets
− Build/Deploy
− MSBUILD, MSI,
BTSTask, Scripts
− Functional test
− BizUnit, Visual Studio
− Load test
− Loadgen, Visual Studio
Building The Lab Environment
Installation and Configuration
• Infrastructure
•
•
−
−
−
−
RAID10/0+1?
Databases and logs separated
Database autogrowth
Virus scanning software
Third party systems
− Non-default values, scale-out options
BizTalk
− Host configuration
− Message tracking
− Adapter configuration
Building The Lab Environment
•
•
•
•
•
Performance monitoring
Deployment
− MSBuild, BTSTask, MSI
Functional testing
Validate automated load test
Document Solution
Performance Baseline
−
−
−
−
Latency
Throughput
Spool depth
etc
Executing the Performance Lab
Assessment High Level Flow
•
•
•
Run automated test
− BizUnit, VSTS, LoadGen
Document results
− KPIs, Check after every run
Tuning for performance
− One modification at a time
− Perform platform
modifications
Session Agenda
− Assessing application performance
− Key learnings
• BizTalk Server performance assessment
− Methodology
− Tools
Helpful Tools
•
•
•
•
•
•
•
•
•
•
BizTalk BPA 1.2
PerfMon
BizTalk LoadGen
BizUnit
Log Man
Relog
Log Parser
PAL
SQLIO
BizTalk Orchestration Profiler
Related Content
•
•
MSDN Guidance
− Microsoft BizTalk Server Performance Optimization Guide
http://msdn.microsoft.com/en-us/library/cc558617.aspx
− Microsoft BizTalk Server Operations Guide
http://msdn.microsoft.com/en-us/library/cc296643.aspx
− BizTalk Server 2006 R2 Hyper-V Guide
http://msdn.microsoft.com/en-us/library/cc768518.aspx
BizTalk Customer Advisory Team Site
− http://msdn.microsoft.com/en-us/biztalk/dd876758.aspx
Resources
•
•
•
•
•
•
•
•
BizTalk Server 2006: Managing a Successful Performance Lab
−
http://msdn2.microsoft.com/en-us/library/aa972201.aspx
Scaling Your Solutions
−
http://msdn2.microsoft.com/en-us/library/aa561567.aspx
Configuration Parameters that Affect Adapter Performance
−
http://msdn2.microsoft.com/en-us/library/aa561380.aspx
Persistence and the Orchestration Engine
−
http://msdn2.microsoft.com/en-us/library/aa547090.aspx
Troubleshooting MessageBox Latency Issues
−
http://msdn2.microsoft.com/en-us/library/aa561709.aspx
Performance Tips and Tricks
−
http://msdn2.microsoft.com/en-us/library/aa560694.aspx
Rule Engine Configuration and Tuning Parameters
−
http://technet.microsoft.com/en-us/library/aa548008.aspx
Performance Considerations When Using the Rule Engine
−
http://msdn2.microsoft.com/en-us/library/aa952929.aspx
Filegroup – Resources
•
•
•
•
•
•
BizTalk Server Database Optimization
−
Understanding Files and Filegroups
−
http://msdn2.microsoft.com/en-us/library/ms189563.aspx
How to: Add Data or Log Files to a Database (SQL Server
Management Studio)
−
http://msdn2.microsoft.com/en-us/library/ms189253.aspx
How to: Move an Existing Index to a Different Filegroup (SQL
Server Management Studio)
−
http://msdn2.microsoft.com/en-us/library/ms175905.aspx
Physical Database Files and Filegroups
−
http://msdn.microsoft.com/en-us/library/ms179316(SQL.90).aspx
Working with tempdb in SQL Server 2005
−
•
http://msdn.microsoft.com/en-us/library/bb743398.aspx
http://www.microsoft.com/technet/prodtechnol/sql/2005/workingwithtempd
b.mspx
Optimizing Server Performance Using I/O Configuration
Options
−
http://msdn2.microsoft.com/en-us/library/ms189892.aspx
MQ Adapter Resources
•
•
HOW TO: Set Up and Test the BizTalk Adapter for
MQSeries
−
http://support.microsoft.com/kb/821489
BizTalk Adapter for WebSphere MQ
−
http://msdn2.microsoft.com/en-us/library/aa772116.aspx
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.