BizTalk Server: Performance Tuning & Optimization - TechNet Articles - United States (English) - TechNet Wiki
Introduction
Performance tuning is a bit more art than science. Not every single tuning technique solves every problem, and very often techniques interact. The approach should always be methodical, including baseline and differential testing particularly when justifying
capital expenditures.
Performance limitations should be viewed as a series of roadblocks. Removing one roadblock doesn’t clear the road, but simply makes it easier to drive. Not until all of the roadblocks have been removed does performance truly jump.
A word of caution. It’s important to understand each tuning technique, both what it does and why it works.
Running application of the included scripts, registry files, etc.
is
not recommended and could lead to serious consequences for the target system. It is not recommended at all to change the settings to the production server directly. It is always recommended to perform the changes
in UAT or DEV and then perform the changes in production in controlled manner following change process of your enterprise.
In this article we are trying to keep everything in one place which are related to performance tuning and optimization for Microsoft BizTalk Server. Most of the concepts are valid across BizTalk Server versions, however it is always recommended to refer
the product documentation of applicable version.
Few Problems
At some point of time, we may face issues in our BizTalk farm and then we think about ways to identify and resolve those issues and we start searching for BizTalk Performance Tuning and Optimization options. Those issues could be one of following or somewhere
similar:
- BizTalk is performing slower,when we drop the messages or during high volume of messages.
- Large no of messages are causing memory errors in BizTalk farm.
- Prolonged BizTalk
Throttling.
- BizTalk receive location is not picking up the files.
- BizTalk Database Jobs are taking too much time to complete.
- Response time of message processing is increasing. There is a high latency in
environment.
- It seems BizTalk is not able to process any message, The messages or service instance are in Active/Running State, however nothing is happening.
- SOAP Adapter is taking long time to call a Web Service.
- Orchestration/Receive Port is taking long time to
debatch a large message.
- The BizTalk production farm has suddenly started behaving in a strange manner.
- High Memory/CPU utilization of the BizTalk servers.
Few Requirements
It is true, at some point of time, we start getting queries, thought or requirement similar to below and then we look for options to scale or optimize the BizTalk Server:
- Looking for creating scalable BizTalk solutions.
- Trying to achieve a particular latency for a BizTalk application.
- Requirement to process 1000 messages in 1 second by a BizTalk application.
- Requirement of processing large messages in BizTalk Server.
- Planning to change the performance throughput.
- Trying to determine/test maximum sustainable throughput
( MSTLooking for creating scalable BizTalk solutions.
- Trying to achieve a particular latency for a BizTalk application.
- Requirement to process 1000 messages in 1 second by a BizTalk application.
- Requirement of processing large messages in BizTalk Server.
- Planning to change the performance throughput.
- Trying to determine/test maximum sustainable throughput
) of a BizTalk farm.
- Trying to figure if BizTalk farm can support next 20+ applications which are going to be deployed in the future.
- How should I decide if
My BizTalk farm needs to be tuned or optimized?
- How should I decide if BizTalk production farm needs to be scaled-up or scaled-out?
What Should We Do
First Few Steps
If you have anything similar in the list above (Few Problems), then we should start with doing below things:
-
Check SQL Server Agent and BizTalk Server Jobs Status of the BizTalk SQL Server Instance - The agent should always be running and all the BizTalk Jobs should be enabled from the day one in your production Farm, If there are any issues refer this in-depth
Wiki
, "
BizTalk Server:Databases:
Survival Guide".
-
The BizTalk Host instance that has the "allow Host tracking" option set is stopped - If the host is stopped, then it should be started without any delay. As a practice there should always one dedicated "
Tracking
Host" should be running in BizTalk Farm. Please refer
"Configuring a Dedicated Tracking Host" MSDN page for more details.
-
Record count of the Spool table- the row count in the tables should not be too much or it should not keep growing. Check the record count of the Host Instance Queue tables.
Refer "
Understanding Host Queue Tables" article for more details.
-
Free space of the Data and log file drives on SQL Server, as there should be enough free space in the log and data files drive on SQL Server.
-
Size of BizTalkDTADb database - Need to make sure the size is not too much of this database, It is a standard practice to keep an eye on the size of this database. Refer the following articles:
-
Excessive Disk I/O Latency (on SQL Server) - If the incoming rate of data into the MessageBox database is faster than which the system can process and move the
data to the BizTalkDTADb database, backlog can build up in the MessageBox database. If the backlog continues to grow unabated, this is a very serious problem and system performance will degrade over time. One way to alleviate this problem is to install faster
disks and/or upgrade the hardware to help ensure that the system is capable of recovering from any message backlogs experienced over time.
Review of Solution Design
Start reviewing the application design and architecture. Sometime a design change may have a huge impact over the application performance. The following should be the starting reference points while reviewing the design and code of BizTalk solutions:
-
Code review of Custom Pipeline component.
-
Are we using XMLDom to load a very large message in
pipeline ?
-
What is the method used to read the values in an incoming message , Use
streaming while reading the message or use XpathReader ?
-
Code of Orchestration?
-
Design
( use of
functoid or inline
xslt) of MAP used in transforming for large messages.
-
Are we using Enterprise Integration Patterns or not while designing enterprise solutions?
-
What kind of Custom Code we are using?
-
Can we use inline send port instead of normal send ports to achieve low latency solutions?
-
How we are calling Business Rule engine in our BizTalk Orchestrations?
- How many scopes have been used in orchestration?
- What kind of instrumentation or tracing is used?
- Use of
BAM API's or
TPE.
References for Code Review/Solution Design
Following articles might be useful while reviewing the solution architecture and the BizTalk Code.
Review Everything Else!
It is also very crucial to review BizTalk environment including SQL and everything else which is your control. The following could be some initial points to start with:
- Hardware configurations of the BizTalk Server Box, and yes its not only about looking RAM and CPU there are other things as well like antivirus settings, hyper-threading etc. It is very important to decide these things while designing the farm. Please see
below reference links for more details.
- Hardware and SAN configuration details of SQL Server instance. Choosing right
RAID level of SAN disks really matters!!
- Network settings can be crucial. Check for instance this article from Sandro Pereira.
- Load balancing of BizTalk Web Services really matters while designing low latency solutions. Need to check options of
IIS tuning.
- Setting BizTalk databases and log file drive size really matters need to review them. See
below references for more details.
References
Below are few references, They have tons of information on performance optimization and tuning of BizTalk.
- (Important!) Microsoft
BizTalk Server 2010 Performance Optimization Guide and BizTalk
Server 2013 Performance Optimization Guide- Always Keep a Copy of this guide, It provides prescriptive guidance on the best practices and techniques that should be followed to optimize BizTalk Server performance. This guide should be consulted while designing
the BizTalk Production or Test Farms.
- Microsoft BizTalk Server 2010 Operations Guide- Provides detailed information for planning a BizTalk Server environment, as
well as recommendations and best practices for configuring, testing, maintaining, monitoring, and optimizing this environment.
- BizTalk Administrator's Checklist Compiled by Microsoft BizTalk Support
- How to troubleshoot SQL Server performance issues
- Performance Tuning Guideline for Windows Server 2008 R2
- BizTalk 2009 – End to end performance testing (1/2)
(
Webcast ) - By Paolo Salvatori
- BizTalk 2009 – End to end performance testing (2/2) (
Webcast ) - By Paolo Salvatori
- (Important!) How
to exploit the Text In R0"> ) - By Paolo Salvatori
- BizTalk 2009 – End to end performance testing (2/2) (
Webcast ) - By Paolo Salvatori
- (Important!)
BizTalk Server Performance: Configuring BizTalk Server for Performance
(
Webcast ) - A must watch webcast to understand the summary of performance tuning with BizTalk Server 2010
- Performance Tuning for Low Latency - A in-depth article about BizTalk Server to achieve Low Latency. If there is a requirement to host low latency
messaging solutions in BizTalk farm, this article might give
lot of information about different parameter changes to achieve it.
- Scaling Out the MessageBox Database -
- Scaling Out the BizTalk Server Tier - Scaling-out is the process of adding additional computers. For example, if BizTalk Server is bottlenecked
by CPU resources, adding another server provides double the CPU resources which may provide double the throughput. You scale out BizTalk Server by adding more computers to your BizTalk Server topology. The following figure shows an example of how you might
scale out the BizTalk tier
- Performance Tips and Tricks - This article provides useful tips for ensuring optimal performance for your BizTalk Server system, It covers following
points in-depth
- Identifying Performance Bottlenecks
- Setting the EPM
Threadpool Size
- Troubleshooting MessageBox Latency Issues
MSDN
-
-
-
-
-
-
-
Throttling - MSDN Blog - Provides details about throttling and how to use performance counters to know more about it.
-
-
-
-
-
-
Processing Large Messages - The article defines large messages and provides guidelines for how to configure BizTalk Server process large messages efficiently
-
-
Evaluating BizTalk Server Performance on Hyper-V -The article describes considerations for evaluating and optimizing performance of a BizTalk Server application running
on a guest operating system installed on a Hyper-V virtual machine. It also provides a description of tools that can be used to measure performance of a BizTalk Server application
-
-
-
Checklist for Analyzing Slow-Running Queries - Since BizTalk database is a highly optimized database and there is no need to further tune the Queries, However sometimes
we use custom databases in our applications, and they could become bottleneck sometime. Refer this article for tuning the SQL queries.
-
BizTalk Server 64-Bit Support - The article provides some frequently asked questions about 64 bit support for BizTalk Server. Refer this article if you are planning to deploy
64 bit BizTalk hosts.
- BizTalk Server 2006 Comparative Adapter Study - This white paper describes the results of a study that compares the performance
of BizTalk Server 2004 and BizTalk Server 2006 adapters
- BizTalk Server 2006: Scalability Case Study Using the SOAP Adapter in BizTalk Server 2006 - This document demonstrates the potential
increase in efficiency through scaled-up server and adapter combinations. Tests were performed on various combinations of BizTalk Servers and SQL Servers. Test criteria, procedures, server and adapter configurations, and results are presented. The data can
help the reader determine which configuration best-suits their existing or anticipated business requirements.
MSDN Social
- MSDN Social Forum for the discussion with Community members - Please refer this forum for posting a new Question or reading about
ongoing discussions.
- MSDN
Social Forum - All threads related to BizTalk Performance Tuning & Optimization
Tools
There are lots of tools described in "Performance optimization guide" by Microsoft and "BizTalk
Server 2010:Tools" wiki, however below are few tools which are very common and it is good to be familiar with them:
-
PAL - Use this tool to gather and analyze performance data from BizTalk, SQL or any other Windows server. This tool provides pre-configured templates with performance counters related
with BizTalk server and SQL Server. The templates can be configured in
perfmon tool to get the performance data, and further data
is used by this tool to generate the performance report.
- Message Box Viewer - A great tool to inspect the BizTalk group. The tool generates a comprehensive report summarizing all the
issues and summary of critical potential problems.
-
- BizTalk Benchmark Wizard- Use this tool to review BizTalk farm after configuring the Farm.
- BizTalk Monitoring Tools - Refer this Wiki to know all the tools to monitor BizTalk Server Farm
- LoadGen - Refer this tool to load generation while performing
load testing on BizTalk Farm
- Process Explorer - The Process Explorer display consists of two sub-windows. The top window always shows a list of the currently active processes,
including the names of their owning accounts, whereas the information displayed in the bottom window depends on the mode that Process Explorer is in: if it is in handle mode you'll see the handles that the process selected in the top window has opened; if
Process Explorer is in DLL mode you'll see the DLLs and memory-mapped files that the process has loaded. Process Explorer also has a powerful search capability that will quickly show you which processes have particular handles opened or DLLs loaded.
- Adapter Trace Utility - Sometime it is required to trace a BizTalk adapter to get the insights of message processing, refer this article for "how to use the
BizTalk Adapter Trace" tools
-
Debugging Tools for Windows - Use Debugging Tools for Windows to debug drivers, applications, and services on Windows systems. Debugging Tools for Windows includes
a core debugging engine and several tools that provide interfaces to the debugging engine. A Visual Studio extension provides a graphical user interface, as does Windows Debugger (WinDbg). Console Debugger (CDB), NT Symbolic Debugger (NTSD), and Kernel Debugger
(KD) provide command line user interfaces.
Blogs
Following are few community blog posts which might be helpful while working with tuning & optimization.
Hot Fixes
Following are few product hot fixes which may be helpful, But it is strongly recommended to read the KB article first and understand if they are absolutely necessary for the product.
- KB2627804 - Hotfix that improves the performance of the tracking feature in BizTalk Server
- KB970070 - Support for Locked Pages on SQL Server 2008 R2 Standard Edition x64, on SQL Server 2005 Standard Edition 64-bit systems, and on SQL Server 2008 Standard
Edition 64-bit systems
- KB944426 - Orphaned cache instances may be built in the Instances and Hosts Queue tables of the BizTalkMsgBoxDb database in BizTalk Server 2006 and in BizTalk
Server 2006 R2
- KB943165 - A CPU usage spike occurs on all the BizTalk hosts after you install security bulletin MS07-040 on a server that is running Microsoft BizTalk Server
2006 or Microsoft BizTalk Server 2006 R2
- kb2387313 - FIX: Poor performance occurs when you use WCF-based BizTalk Adapter for Oracle Database on a dynamic port that is used by BizTalk Server
See Also
Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is
BizTalk Server Resources on the TechNet Wiki.
