Stories from the field Darren Jefford Solution Architect, Microsoft UK [email protected] Some Topics.. Suitability? PoC Latency Throttling ESB BizTalk Architect Non Functional Requirements Maintability, Manageability, Scalability Operations/Administration Instrumentation.
Download ReportTranscript Stories from the field Darren Jefford Solution Architect, Microsoft UK [email protected] Some Topics.. Suitability? PoC Latency Throttling ESB BizTalk Architect Non Functional Requirements Maintability, Manageability, Scalability Operations/Administration Instrumentation.
Stories from the field Darren Jefford Solution Architect, Microsoft UK [email protected] Some Topics.. Suitability? PoC Latency Throttling ESB BizTalk Architect Non Functional Requirements Maintability, Manageability, Scalability Operations/Administration Instrumentation Scenario 1: App to App integration BizTalk Server BizTalk Server Central system Complex translations Adapter support Scalability (perf & endpoints) Management, Audit Departmental System(s) Infrastructure dependencies Management overhead Scenario 2: B2B Integration BizTalk Server BizTalk Server Organisation A Organisation B Shared network · Messages need to be reliable · Message need to be secure · Shared network maybe volatile Volatile network support Rich & flexible security Adapter support Tracking & Audit Infrastructure dependencies Management overhead Scenario 3: “Message Bus” BizTalk Server Integration Platform · · · · · Diverse connected systems Any system may communicate with any other High volume messaging Standards driven Monitoring & audit requirements Publish subscribe model Message routing Loose coupling Standards, governance Infrastructure dependencies Loosely coupled Standards, governance Scenario 4: Business Processes BizTalk Server Fulfillment Business Processes “Cancel” Customer Site Delivery Step 1 Step 2 Step 3 Step 4 Step 5 CRM System Invoicing · Business processes involving multiple systems · Different systems have different availability · Business Processes can be long lived Resilience Tooling Scalability Enterprise class Human Workflow Infrastructure dependencies Management cost Why do I need BizTalk? “Using all the Microsoft integration technologies, could I not build the solution without BizTalk Server?” “Yes. But………” Thread Management Scripting & Programmability APIs Memory Management Service Isolation Exception Configuration Regulatory Compliance Failed Message Management Archiving & Purging Message Tracking Disaster Recovery Utilization/Performance Tracking Composite Execution Tracking Health Monitoring Composite State Management Load Balancing Availability Monitoring Multi-environment Deployment Model Fail-over Configuration Management Identity & Impersonation Throttling Scale-out Configuration Managing complexity Complexity of solution Custom solution Scenario 1 Scenario 2 Complexity of requirements Scenario 3 BizTalk environment considerations Management skills • Good or bad management will make or break your BizTalk solutions reputation BizTalk Server is a generic platform Is the customer stretching it to an edge case? BizTalk Performance & operational factors Setting performance and capacity expectations Know the factors that affect capacity Understand the expected message latency Our model has both strengths & weaknesses Transactional integrity is provided by your transports How do you answer the “Will I ever loose data?” question COM+ / Ent. Svcs / Remotin g Role based security Legacy connectivity Transactional support WCF Application to application Low level control of transports Support for WS standards Inter-server comms Complex to configure Not firewalls friendly Specialist approach e.g. Transact, Reliable etc One to many connectivity not easy No standardised monitoring & audit Queu es SSIS / DTS BizTalk Server Disconnected / volatile networks Database to DB connectivity Ordered Delivery Very large volumes of data Transactional transports Batch data transfers DB queuing (SSIB) Tightly coupling of data Scalability of performance Scalability of requirements Scalability of transports Standard management "Smoothing" load Data warehousing / Bus Intelligence Adapter model Creating Sync over async facades "Single customer view" scenarios Business Process Legacy modernisation High volume data enrichment Prioritisation Many to one data merge Infrastructure requirements Asynchronous programming model Organisational commitment to queues Singleton message transfer Loose coupling of system Abstract / Application interfaces Resilience Standardised Audit Standards enforcement BizTalk Server developers skills Infrastructure requirements Very low latency considerations Very large messaging Message Agent Subscriptions MessageBox Adapter Pipeline Pipeline Orchestration Pipeline Outbound Pipeline Pipeline Pipeline Inbound Adapter Adapter Adapter Adapter Adapter Pub/Sub with Orchestration “Receive” Throttling Receive Throttling relates to Throttling imposed on the receipt of messages into the BizTalk Message Box Applies to hosts containing Adapters Orchestrations (that create and publish messages to the Message Box) Message publishing throttling state performance counter Implemented through EPM Delays, EPM Threads Publishing throttling states 0: No throttling 2: Rate 4: Process Memory 5: System Memory 6: Database size 8: Database session count 9: Thread count 11: User Override Processing Throttling Processing Throttling relates to Throttling imposed on the processing or consumption of messages from the BizTalk Message Box Applies to hosts containing Send Ports and Orchestrations Message delivery throttling state Implemented through Delay in delivering messages In-memory queue size is reduced (100) Thread Pool reduced Delivery throttling states 0: No throttling 1: Rate 3: Unprocessed message 4: Process Memory 5: System Memory 9: High Thread Count 10: User Override Common Symptoms “Orchestration completed/sec drops off suddenly” Not enough work? SQL Server problems Throttling Common Symptoms.. “High CPU” Rules Engine Translation Orchestration serialization Persistence points Custom Code Serialization, System.Xml, RegEx, etc Visual Studio Profiler is your friend Common Symptoms.. “High Memory Usage” Rules Engine cache BizTalk suppresses GC Throttling will kick in after ~600MB Common Symptoms.. “There were not enough free threads in the ThreadPool object to complete the operation” “Web Service calls performing slowly” Thread Starvation All in one host? Too slow to create threads? Common Symptoms.. “Low Throughput (not achieving the msg/sec you want)” Are you feeding BizTalk with enough work? Inspect Spool and Host queues Is BizTalk throttling? Dependant system keeping up? Poor performing disks Extra BizTalk servers if throttling is not occuriong, sql is looking good Common Symptoms.. “Heavy BAM Usage” BAM_DM_<ActivityName> SSIS job Tracking Data Size perf counter Common Symptoms.. “Spool increasing steadily” Inspect Host Queues, which one building, rx, ox or tx? Host throttling? Check SQL Server Agent is running?