Transcript Charlie Chung Lead Program Manager Microsoft Session Code: UNC311 Session Objectives And Takeaways Session Objective(s): Describe new High Availability and Service Level Reporting features of the.

Charlie Chung
Lead Program Manager
Microsoft
Session Code: UNC311
Session Objectives And Takeaways
Session Objective(s):
Describe new High Availability and Service Level
Reporting features of the Exchange Server 2010
transport platform
Explain how to deploy Exchange Server 2010
transport server including coexistence with
Exchange Server 2007 and Exchange Server 2003
Deploy highly available transport designs that
deliver messages with low latency
Understand key coexistence scenarios
Agenda
Exchange Server 2010 Transport Overview
New Transport High Availability Features
Managing and Reporting Transport SLA
Exchange 2010 Routing overview
Interoperability and coexistence with Exchange
Server 2003 and 2007
Exchange 2010 EdgeSync Enhancements
Exchange Server 2010
System Architecture
Edge Transport
Routing & AV/AS
Enterprise Network
Hub Transport
Routing & Policy
Active
Directory
External
SMTP
servers
Mobile phone
Web browser
Mailbox
Storage of mailbox
items
Unified Messaging
Voice mail &
voice access
Client Access
Client connectivity
Web services
Phone system
(PBX or VOIP)
Outlook
(remote user)
Outlook (local user)
Line of business application
Exchange Server 2010
Hub Transport Role Architecture
RPC
From:
To:
Hub Transport
Mailbox
1. User composes message in
Outlook and it is stored in users
Outbox
2. Mailbox submission service
listens for store event
notification of new message and
notifies an in-site Hub Transport
3. Hub Transport retrieves
message from sender’s mailbox
and submits to queue
4. Hub Transport categorizes
message and applies message
policies
6. Hub Transport delivers
message to mailbox server in
same AD site
Mailbox
5. Hub Transport delivers
message to Hub Transport
server in target AD site
Hub Transport
Transport High Availability Architecture
Resiliency Issues in Exchange Server 2007
Transport database is stateful
Loss of service results in loss of mail
Hardware redundancy for high availability
Transport dumpster impacts the environment
In extreme cases, up to 200% increase in IOPS/message due
to many SGs and inefficient cache usage when compared to
similar scenarios without dumpster
Redelivery after MDB failover results in entire quota being
redelivered and store removing duplicates
Transport database corruption causes downtime
Mail storms due to rogue user/program
Transport High Availability Architecture
Exchange 2010 Resiliency Improvements
Shadow Redundancy is a new feature of Edge and Hub
transport roles
Provides redundancy for messages in transit
Transport becomes near-stateless
Eliminates need for RAID1/10 storage for queue database 
50% write I/O is eliminated
Enabled by default
Transport resilient to database corruption
Will move/delete old database and restart service
Throttling of MAPI and SMTP client submissions
Prevent mail storms due to accidental misuse, misbehaving
software and malware
How does Shadow Redundancy Work?
1. Hub (shadow) delivers message to Edge1
(primary)
Detects that Edge1 supports Transport
redundancy through XSHADOW verb
Hub moves message to shadow queue and
stamps Edge1 as current, primary owner
Hub
1
Edge1
Edge2
2
Foreign
MTA
2. Edge1 (primary) receives message
(becomes “primary owner”)
Edge1 delivers message to next hop
Edge1 updates discard status of the
message indicating delivery complete
to foreign MTA
How does Shadow Redundancy Work?
Hub
1
4
3
3. Success: Hub (shadow) queries Edge1 (primary) for
expiry status
Hub issues XQDISCARD command (next SMTP
Session),Edge1 checks local discard status and
responds with list of messages considered delivered
 Hub deletes messages from its shadow queue
4.
Edge1
Edge2
2
Foreign
MTA
Failure: Hub (shadow) queries Edge1 (primary)
discard status and resubmits
Hub opens SMTP session, issues XQDISCARD command
(heartbeat)—if Hub can’t contact Edge1 within 15
minutes (3X timeout interval), resubmits messages in
shadow queue—resubmitted messages are delivered
to Edge2 (go to #1)
Shadow Redundancy
Primary Server State Tracking
Shadow server needs to determine Identity of Primary Server
If identity change detected, shadow messages for primary are
resubmitted
“Heartbeat” needed to determine when shadow server should
resubmit shadow messages for delivery over alternate route
Failure to complete successful heartbeat results in resubmission of
shadow messages (default 3 attempts at 5 min interval)
“Discard Status” needed to determine when shadow server can
delete shadow message after delivery completed
At end of each SMTP session, shadow server issues XQDISCARD
command which returns list of unique ID’s that can be removed from
shadow queue
Shadow Redundancy
Supported Scenarios
4
5
3
SMTP
Client
Hub
Mailbox
6
2
5
2
1
5
3
Mailbox
Hub
4
0
Client
Edge
Ex2007
Hub
6
4
Internet
1) Mail Submission Service
MSExchangeMailSubmission saves shadow message
copy in sender’s “Sent Items” folder, critical properties
of message are hashed to ensure it is valid for
resubmission
“Implicit” heartbeat piggybacks on RPC (Remote Procedure
Call) notification used for store driver submission
“Explicit” heartbeat invokes extra RPC in absence of store
driver submissions
Shadow message discard status also piggybacks on MSRPC
used for store driver submission
Remaining shadow message(s) resubmitted from “Sent
Items” after 3 explicit heartbeat failures
Shadow Redundancy
2) SMTP Service Extensions
New SMTP service extensions
XSHADOW
XQDISCARD
Used to provide redundancy between Exchange 2010
transport servers over SMTP
Intra-Forest message transfer using Exchange Servers
authentication (Hub-Hub, Hub-Edge)
Cross-Forest message transfer using externally secured send
and receive connections
Saves copy of message on previous hop until next hop
fully delivers all recipients
Shadow Redundancy
XSHADOW Configuration
Organization Configuration (*-TransportConfig)
ShadowRedundancyEnabled
ShadowHeartbeatRetryCount
ShadowHeartbeatTimeoutInterval
ShadowMessageAutoDiscardInterval
:
:
:
:
True
3
00:05:00
2.00:00:00
Receive Connector Configuration
Authentication Mechanisms enable advertisement of SMTP
service extensions
Exchange Servers
Externally Secured
Permissions enables client to use commands
ms-Exch-SMTP-Accept-Xshadow
Send Connector Configuration
Permissions enable use of commands
ms-Exch-SMTP-Send-XShadow
Shadow Redundancy
SMTP Session with “Implicit Heartbeat”
< 220 PRIMARY.TEST.COM Microsoft ESMTP MAIL Service ready at Tue, 4 Sep 2007 10:07:15 -0700
> EHLO SHADOW.TEST.COM
< 250-PRIMARY.TEST.COM Hello [10.197.93.136]
< 250 XSHADOW
> XSHADOW FzHkA/yKi0GHWQnBHzdbOg==
< 250 VUjDMdghpkm4OwsLyqZcag==
> MAIL FROM:<[email protected]> SIZE=1005 XSHADOW=e21e97f4-f911-47d5-99aa-6b3c8757f73b
> RCPT TO:<[email protected]>
< 250 2.1.0 Sender OK
< 250 2.1.5 Recipient OK
> BDAT 1336 LAST
< 250 2.6.0 <[email protected]> Queued mail for delivery
> XQDISCARD 50
< 251 OK, no discard events
> QUIT
< 221 2.0.0 Service closing transmission channel
SMTP Session with “Explicit Heartbeat”
< 220 PRIMARY.TEST.COM Microsoft ESMTP MAIL Service ready at Tue, 4 Sep 2007 10:12:27 -0700
> EHLO SHADOW.TEST.COM
< 250-PRIMARY.TEST.COM Hello [10.197.93.136]
< 250 XSHADOW
> XSHADOW FzHkA/yKi0GHWQnBHzdbOg==
< 250 VUjDMdghpkm4OwsLyqZcag==
> XQDISCARD 50
< 250 e21e97f4-f911-47d5-99aa-6b3c8757f73b
> QUIT
< 221 2.0.0 Service closing transmission channel
Queue Viewer
Shadow Queue
Queue Viewer
Shadow Message
3) Mailbox Delivery
Transport Dumpster continues to provides redundancy for final
delivery to mailbox
ActiveManager provides MDB replication feedback to transport ,
used to control which messages are retained in the
Transport Dumpster
When log containing delivered message has been replicated to all MDB
copies, message is truncated from Transport Dumpster
Dumpster size is now a function of MDB log replication latency
and frequency of feedback, maximum size limited by quota
when one or more MDB copies not healthy
Mailbox Role requests re-delivery from all hub servers in all AD
sites hosting copy of MDB after cross-site failover
Shadow Redundancy
4) Delayed Acknowledgement
“Best Effort” shadow redundancy for any SMTP implementation that doesn’t
support XSHADOW and XQDISCARD
No shadow redundancy for outgoing messages to these systems
Delayed Acknowledgement after end of data sequence
250 response delayed up to 30 sec (default) while categorization and delivery
are attempted
If transport server fails before acknowledgement, client resubmits
Message will “skip” the delayed ack when DelayedAckSkippingEnabled is
true and any of the following conditions exist:
Submission queue in suspended state
Message is deferred due to transient error
Delivery queue in retry or suspended state
Delivery queue size exceeds DelayedAckSkippingQueueLength value defined in
EdgeTransport.exe.config (default 100)
Message routed to unreachable queue
Delayed Acknowledgement Configuration
Organization Configuration (*-TransportConfig)
ShadowRedundancyEnabled
Receive Connector Configuration
MaxAcknowledgementDelay
Default 30 seconds
Disable by setting to 0 seconds
Do not exceed 60 seconds for client connector
Do not exceed 10 minutes for default connector
EdgeTransport.exe.config
DelayedAckSkippingEnabled
DelayedAckSkippingQueueLength
5) Side Effect Messages
System generated messages (Journal Report, NDR) are
considered “side effects” of original message
submission
Resubmission of shadow message copy will occur if
“primary” and any associated “side effect” messages
are not delivered before server failure
Resubmission of shadow message copy will result in
the same “side effect” messages as the original
message
Diagnostics
Message Tracking Log RESUBMIT events indicate when
messages are resubmitted due to shadow redundancy heartbeat
failure or transport dumpster redelivery
SMTP Receive Protocol log provides info events for delayed
acknowledgement including reason for DelayAck skipping
MSExchangeTransport Shadow Redundancy Perfmon object
“Current Messages Acknowledged Before Relay Completed” provides
count of messages accepted without redundancy
Events indicate when transport receives redelivery requests
from mailbox role for each MDB after failover, when
resubmission job is completed and how many messages were
resubmitted by transport from transport dumpster
Queue Database Resiliency
Automated Recovery
Transport detects fatal ESE exceptions associated with
Queue database
Moves or Deletes database
Default to move (requires manual action before subsequent
recoveries are attempted)
Optionally enable delete action in app.config (no manual
operation necessary unless failure occurs)
Service process restarts worker process
New Queue database created
Method not always successful
Hardware failures (drive, controller, etc) require manual
recovery actions
Throttling Message Submissions
Manage using *-ThrottlingPolicy cmdlets
Throttling policies are applied per-user
Transport settings in Default Throttling policy are disabled by default
Default Policy can be overridden with custom policy applied to individual users
MessageRateLimit throttles rate of message submission from
authenticated user or anonymous IP address
Evaluated per-server over 1 minute period
SMTP returns transient errors when rate exceeded
Mail Submission Service defers messages in outbox once rate has been exceeded,
retries submission periodically
RecipientRateLimit throttles number of messages submitted
Evaluated over 24 hour period
Central accounting on mailbox role using MSExchangeThrottling service
Error returned to client for all submission attempts once quota exceeded
Transport Service Level Management
Monitoring, Incident Management and Reporting
Key Heath Indicators: Message Latency, Availability
Service Level Metrics
Noise
Gaps
Scope/Impact/Expertise
HA is mitigation
Alert the right person
Capacity Planning
End User Experience
Reporting
Alert when Service Level Threatened
Diagnosis
Root Cause Analysis (% identified)
Instrumentation and Analysis Tools
Recovery
Mean Time to Recovery (MTTR)
Self-Healing
Standardized Recovery Process
Processes that impact ability to
meet SLA objectives
Performance against SLA
objectives
Awareness
Transport Service Level Management
Awareness through Proactive Monitoring
Key Health Indicators (KHI) used to determine
when user experience impacted
Delivery Latency to determine if delivered
messages are meeting SLA objectives
Submission Availability to determine if server is
available to accept new messages
DSN Generation to determine if server is failing to
deliver messages
Delivery Completion to determine if server is
unable to complete delivery
Transport Service Level Management
Measuring Delivery Latency
Exchange Server 2010 measures latency of every component
involved with delivering message end-to-end
Previous Hop latency using Received Headers timestamps for
measuring delivery latency on legacy transport servers
Define IP ranges using InternalSmtpServers parameter on transport
configuration (*-TransportConfig)
Recommend NTP for accurate measurements
get-message cmdlet has new IncludeLatencyComponent
parameter to determine latency of message in queue
“MSExchangeTransport Component Latency” Perfmon object
counters for local server percentile latency measurements over
moving 5 minute window
End-to-End latency of “delivered” messages can be determined
from message tracking logs on final hub
Measuring Delivery Latency
Message Tracking Log Details
[PS] C:\>get-messagetrackinglog –server:df-mlt-01 -messageid:
<[email protected]>" | ConvertToMessageLatency.ps1 | FT -a ComponentServerFqdn,ComponentCode,ComponentName,ComponentLatency
ComponentServerFqdn
ComponentLatency
------------------msw-sfw-r03.redmond.corp.microsoft.com
tk5-exsmh-c102.redmond.corp.microsoft.com
tk5-exhub-c103.redmond.corp.microsoft.com
TK5EX14MLTC101.redmond.corp.microsoft.com
df-h14-01.exchange.corp.microsoft.com
DF-MLT-01.exchange.corp.microsoft.com
ComponentCode ComponentName
------------- ------------TOTAL
TOTAL
TOTAL
TOTAL
TOTAL
TOTAL
Total
Total
Total
Total
Total
Total
Server
Server
Server
Server
Server
Server
--------------Latency
Latency
Latency
Latency
Latency
Latency
00:00:03
00:00:23
00:00:08
00:00:00
00:00:00
00:00:00
Hop 1: 3rd Party Application MTA (Previous Hop Latency)
Hops 2,3: Exchange Server 2007 (Previous Hop Latency)
Hops 4,5,6: Exchange Server 2010 (Latency Tracker)
End-to-End Delivery
Latency of ~34
seconds
Measuring Transport Service Levels
System Center Aggregation and Reporting
Primary Datacenter
Location
Edge
Hub1
Hub2
System Center
Root Management Server
Remote Locations
SQL Reporting SQL Server
Services
Database
Measuring Transport Service Levels
Statistics Log Generation
Server statistics log generated hourly (00:00-23:00)
containing traffic summary
ServerStatisticsLogMaxAge
: 30.00:00:00
ServerStatisticsLogMaxDirectorySize
: 250 MB (262,144,000 bytes)
ServerStatisticsLogMaxFileSize
: 10 MB (10,485,760 bytes)
ServerStatisticsLogPath
: C:\Program
Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\ServerStats
Active user statistics log generated every 8 hours
(00:00, 08:00, 16:00) containing summary of user
usage
ActiveUserStatisticsLogMaxAge
: 30.00:00:00
ActiveUserStatisticsLogMaxDirectorySize : 250 MB (262,144,000 bytes)
ActiveUserStatisticsLogMaxFileSize
: 10 MB (10,485,760 bytes)
ActiveUserStatisticsLogPath
: C:\Program
Files\Microsoft\Exchange
Server\V14\TransportRoles\Logs\ActiveUsersStats
Exchange Server 2010 Routing
Few changes from Exchange 2007 routing architecture
Direct connections (point-to-point routing)
Prefer direct IP connection between source and destination
Based on AD site topology and site link costs
Queue mail as close to destination as possible
Deterministic routing
Simplify design to follow a consistent pattern make planning and
troubleshooting easier
No longer relies on Exchange Link State information
Optimize bytes over the wire by bifurcating based on route
Simplify deployment
Automatic configuration
Consolidated topology concepts
Active Directory Sites Are
The Routing Boundary
Automatic load balancing and fault tolerance
Mailbox will load balance submissions across all Hubs in local AD site
When mailbox and Hub roles coexist on same server, local Hub preferred
Hub will load balance connections across all Hubs in remote AD Site
Hub will deliver to any mailbox in local AD site
Uses the AD site topology to calculate back-off
Direct connect FIRST, unless forced through Hub Sites
Provides for queuing at the point of failure
Availability information is not cached
Always try all Hub servers within remote AD site before back-off
Each new connection uses same algorithm
When bifurcation (delayed fan-out) is required
Equal cost path arbitration
Hop count
Alphabetic based upon site name
“Best” Route Between AD Sites
Final Backoff
Direct Connect
Site 11
Backoff Route #2 Backoff Route #1
10
0
Site 2
Site 1
Cost = 100
st
Co
00
=
=1
Co
st
=
Co
st
Originator
Site 21
10
0
Site 3
s
Co
t=
10
0
Recipient #1
Coexistence with Exchange Server 2003
All Exchange 2007/2010 servers are within a single
routing group
Introduction of first Exchange 2007/2010 Hub role
results in creation of routing group connectors (single
source/target bridgehead on each)
Add source and target bridgehead servers for fault tolerance
and load balancing between these two connected routing
groups
Exchange 2003 RGC bridgehead cannot be a cluster
Coexistence with Exchange Server 2003
Exchange 2007/2010 Routing to Exchange 2000/2003 recipient
Chooses least cost RGC route to Exchange 2003 recipient based on
routing group connector costs (AD cost not included)
Chooses least cost route within the Exchange 2007/2010 routing group
to the AD site containing RGC “bridgehead” based upon AD site link cost
Exchange 2000/2003 routing to Exchange 2007 recipient
Server picks least cost route to the Exchange 2007/2010 Routing
Group regardless of AD site where recipient mailbox located
Exchange 2007/2010 “bridgehead” routes within Exchange 2007/2010
Routing Group to the AD site containing recipient mailbox based upon
AD site link cost
Exchange 2010 Transition Topology
Site 11
Originator
E2010
Co
st
=
10
0
Site 2
Site 1
Bifurcate
E2010
Cost = 100
E2010
=
st
Co
Site 13
E2010
E2010
E2010
E2010
Co
st
=1
00
RGC
Cost=10
Exchange 2003
Routing Group 13
Routing Group
Connector
(RGC)
Cost=10
RGC
Cost=10
Site 23
E2010
Exchange Routing Group
(DWBGZMFD01QNBJR)
RGC
Cost=10
Recipient #2
1
00
RGC
Cost=10
RGC
Cost=10
RGC
Cost=10
Exchange 2003
Exchange 2003
DisableRouting
Link
State
on all E2K/E2K3
Servers!!!
2
Routing Group
1
Group
Exchange 2003
Routing Group 23
Recipient #1
Disabling Link State
Suppresses communication of minor
link state changes (link up or down)
Used when you have multiple routes to/from
the Exchange 2010/2007 Routing Group
Must be done to every Exchange 2003 server in
the organization to prevent loops
All versions only use least cost route
Controlled via registry
HKLM\System\CurrentControlSet\Services\RESvc\Parameters
DWORD: SuppressStateChanges
Value: 1
RPC
From:
To:
Hub Transport
Mailbox
1. User composes message in
Outlook and it is stored in
users Outbox
2. Exchange 2007 Mailbox
submission service listens for
store event notification of
new message and notifies an
in-site Exchange 2007 Hub
Transport server
6. Exchange 2010 Hub Transport
delivers message to Exchange 2010
mailbox server in same AD site
3. Exchange 2007 Hub Transport
retrieves message from sender’s
mailbox and submits to queue,
categorizes message, applies
4.
Exchange
2007
Huband
Transport
Exchange
2007
policy
drops
delivers
message
to Exchange
in “Version
14” delivery
queue
5.
Exchange
2010 Hub
Transport
2010
Hub Transport
server
in same
receives
message
AD site using
SMTPvia SMTP,
categorizes message, applies
Exchange 2010 policy, queues to
Exchange 2010 mailbox server
Mailbox
Hub Transport
Coexistence with Exchange Server 2007
Routing version boundary change:
Exchange 2010 Mailbox servers can only submit to Exchange 2010 Hub
Transport servers
Exchange 2010 Hub Transport servers can only deliver to Exchange 2010
Mailbox servers
Exchange 2007 Mailbox servers can only submit to Exchange 2007 Hub
Transport servers
Exchange 2007 Hub Transport servers can only deliver to Exchange 2007
Mailbox servers
Exchange 2010 Hub Transport servers can communicate with Exchange 2007
Hub Transport servers via SMTP (and vice versa)
Inter-site routing has no version preference
Hub role will load-balance inter-site traffic to all hubs in target site
Subscribed Edge servers:
Have no version preference when routing inbound/outbound traffic
Exchange 2010 Hub Transport will become authoritative for Edgesync
Edge Transport Role
EdgeSync Improvements
Better Performance for EdgeSync via Deltasync Mode
Under this mode, each time EdgeSync service only reads
the delta change since last sync and updates the
target accordingly
Support for safe senders and blocked senders
Configurable Safe List quotas
Administrator defined blocked senders
Automatic update of Safe Sender list propagation into
Active Directory
Key Learnings
Understand how New Transport High
Availability and Service Level Reporting features
of the Exchange Server 2010 can lower the
capex and opex costs for Hub Servers
Understand how Exchange Server 2010 mail
routing coexistence works with Exchange Server
2007 and Exchange Server 2003 so you can plan
your upgrade
Aware of the new instrumentation, tools, and
reports for you to measure the SLA of mail flow
in your environment.
UNC Track Call to Action!
Learn More!
Related Content at TechEd on “Related Content” Slide
Attend in-person or consume post-event at TechEd Online
Check out learning/training resources at Microsoft TechNet
Exchange Server and Office Communications Server
Check out Exchange Server 2010 at
Virtual Launch Experience (VLE) at thenewefficiency.com
Try It Out!
Download the Exchange Server 2010 Trial
Take a simple Web-based test drive of UC solutions through
the 60-Day Virtual Experience
Resources
www.microsoft.com/teched
www.microsoft.com/learning
Sessions On-Demand & Community
Microsoft Certification & Training Resources
http://microsoft.com/technet
http://microsoft.com/msdn
Resources for IT Professionals
Resources for Developers
Complete an evaluation
on CommNet and enter to
win an Xbox 360 Elite!
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.