Microsoft Exchange 2013 Sizing – An Information Behind the Guidance

About Me – Charles Derber

 Have been working on Microsoft Messaging for ALMOST a Decade  Experienced on Planning / Implementing / Migrations / Designing / Consulting for Messaging Enterprise & Hosting Solutions  Also experienced on Multiple Messaging Vendor like Microsoft, IBM, Linux etc. but gained much expertise in “Exchange” due to exposure and interest on Microsoft Products  Speaker @Mumbai IT Professional User Group & I do blog @ http://charlesgate86.wordpress.com

Agenda

       In the beginning – Road to Sizing Architectural Changes Impact on sizing Sizing Server role specific Deep drive - MBX / CAS / Active Directory Exchange Roles Sizing – Storage / Networks / Virtualization / Hardware Exchange 2013 Sizing tools Considering Exchange 2013 Virtualization Sizing – Demo

Sizing is both a “Science” & an “Art” form.

Guidance  Where it is coming from? – Pre-release environment!!

 (Test labs / Dog Food(emails are used in a way where it is not intended to use / Yes multi profile) / MSIT / Customer / Field work) Resources   TechNet / MS Exchange Team Blog / Experts Blogs Third Party Tools

What is not possible…?

 Doesn’t cover everything because it cannot cover everything - but will provide some base line to start with   Third party solution - BB / SMTP Relay / Applications  Third Partner dependent - Every one has different architect / functionality  Hardware support  Needs lab testing for your environment / consider failure domains - High Availability

6. Finalize Design 6 STEPS SIZING PROCESS 1. Read Guidance 5. Consider Impact 2. Collect Data 4. Input Profile Data 3. Define Constraints

Sizing Process

1. Read / Understand sizing, scalability, capacity guidance  Documentation @TechNet / Exchange Team Blog 2. Collect Data on existing deployment   User profiles (message sent + received per day) Average message size

Sizing Process

3. Define constraint        No of database copies Backup requirement Storage architecture Safety Net duration Virtualization Growth Plans 3 rd Party Products

Sizing Process

4. Input profile data and design constraints into calculator tool  Always use latest calculator 5. Consider impact of various option provided by results via Calc.

   Cost Rebuilt times Impact on High Availability

Sizing Process

6. Finalize Design   Storage calculator provides configuration scripts Archive the calculator as documentation of the sizing process

Architectural Changes / Features

Exchange Server Roles CAS Stateless – OWA / OAB / OA / EAS / EWS / Mail Flow MBX – PF HA / Managed Store / IOPS / Safety Net & Others Balanced use of Hardware – CPU / Memory / Disk

Minimum Systems Requirement



RAM

8GB – Mailbox or Multirole/Mailbox+CAS 4GB – CAS 

CPU

Follow published OS guidelines 

Disk Space

30GB

RAID vs JBOD, DAS vs SAN, Oh My!

Where is your personal/precious email – JBOD     Google email account is stored without RAID Microsoft's Office 365 cloud service utilizes JBOD for storage Microsoft's own internal deployment of Exchange 2010 has over 180,000 users on a JBOD infrastructure Microsoft's TerraServer uses JBOD to store all of the data presented through that web site  Microsoft therefore recommends that if you are considering utilization of drives in a JBOD architecture that you do choose to do so with the 7.2K RPM SAS drives rather than SATA

What Storage should I choose ?

 Even I can't tell you which storage architecture to choose  Microsoft supports SAN and DAS+RAID along with DAS+JBOD because no single storage architecture is right for every customer out there.

 Unless you are in a situation where money is no object and will continue to be no object, you must seriously evaluate the storage solutions available to you

Sizing Exchange 2013 – Mailbox Role

Mailbox Role - Impact

 Increased resource usage  Cache sizing is different  New content indexing architecture impacts performance  By default UM enabled on MBX  Managed availability

Mailbox Role - Storage

 Mailbox Database, content indexes, log space  20% database overhead is now 0%  Content Index size is now 20% of EDB  Plus space for additional index set per volume (master merge)  Note that impact of space for master merge is reduced with multiple DBs per-volume

Mailbox Role – Storage Bandwidth

 BDM in 2013 now consuming ~1MB/sec/DB copy, significant reduction from 2010

Mailbox Role – Transport Storage

 Transport queue database takes advantage of ESE IO improvements to reduce IOPS  Microsoft production observations show ~1 DB IO per 75KB message  Low IOPS suggest that placing transport queue on system/install volume is now feasible in many scenarios  Significant transport throughput benefit seen from a protected write cache disk controller, set to 100% write cache  Transport capacity requirements include queue and Safety Net

Mailbox Role – IOPS

 Exchange 2013 reduced IOPS requirements (~33% reduction compared to 2010)  Checkpoint depth is now consistent for all DB copies, so IOPS requirements are the same

Mailbox Role – CPU

Messages sent or received per mailbox per day

50 100 150 200 250 300 350 400 450 500

Mcycles per User, Active DB Copy or Standalone (MBX only)

2.13

4.25

6.38

8.50

10.63

12.75

14.88

17.00

19.13

21.25

Mcycles per User, Active DB Copy or Standalone (Multi Role)

2.66

5.31

7.97

10.63

13.28

15.94

18.59

21.25

23.91

26.56

Mcycles per User, Passive DB Copy

0.69

1.37

2.06

2.74

3.43

4.11

4.80

5.48

6.17

6.85

Note: Baseline platform for CPU guidance changed in 2013. Mcycle requirements in 2010 and 2013 cannot be directly compared.

Mailbox Role – CPU

 Hyperthreading (SMT) – Fantastic Technology increases CPU throughput – DONT USE IT…!

 .Net framework allocates memory per core heaps  Significant impact to some Exchange service memory footprints

Mailbox Role – Memory

 Memory on the Mailbox role sized based on ESE cache requirements  Cache requirements have remained constant from 2010  Overall cache sized to 25% of RAM, so guidance (based on total system memory) is 4x of 2010 cache sizing recommendation  Multi-role servers require additional memory for CAS based on user concurrency during worst-case failure

Messages sent or received per mailbox per day Mailbox role memory per active mailbox (MB)

50 100 150 200 250 300 350 400 450 500 12 24 36 48 60 72 84 96 108 120

Mailbox Role – Memory

 Memory is allocated within heaps  .NET garbage collector has different “modes” which optimize for different allocation scenarios  Workstation GC  Uses common heap and cleanup process (can be concurrent or not)  Server GC  Allocates a heap and thread per logical proc  Server GC results in dramatically larger memory requirements at rest when SMT is enabled

Mailbox Role – Memory

 Best practice to install KB 2803754 or 2803755  http://support.microsoft.com/kb/2803754  http://support.microsoft.com/kb/2803755 (Windows Server 2008 R2) (Windows Server 2012)  Reduces memory consumption in each store worker No impact to sizing guidance Memory is available for use by other processes  Decreases CPU spent in .NET garbage collector  Benefits Mailbox & multi-role  Enable by installing hotfix and setting regkey: HKLM\Software\Microsoft\.NETFramework\DisableRetStructPinning (REG_DWORD) = 1

Mailbox Role – Network

 Seeding of multiple database per disk may require increased bandwidth between servers  Avoid bottlenecking on network  Plan for reseed operations, particularly in JBOD deployments  10GB Ethernet expected to become more common for Exchange infrastructure  Cost has dropped, many customers are standardizing on 10GB Ethernet in their datacenters

Mailbox Role – Unified Messaging

 UM is sized using a concurrency model – you cannot have a dedicated server role as UM  Plan for a maximum of 100 concurrent calls per server  Note that UM in 2013 is a component of the Mailbox role, may need to adjust user distribution to optimize UM utilization/concurrency  Voicemail transcription is a heavy consumer of CPU  Plan for 1 CPU core per concurrent transcription If server is CPU starved, voicemail transcription may be skipped (voicemail delivered without transcription)

Sizing Exchange 2013 – CAS Role

CAS Role - Impact

 Low resource utilization makes multi-role deployment (or virtualization) attractive  CAS is intelligent director of traffic work  Shift of processing resources from LB layer to CAS may negate new resource demand

CAS Role – Processor

 CAS CPU is sized using a percentage of Mailbox CPU active user requirements  2013 CAS requires 25% of Mailbox active-user mcycles, down from 75% in 2010  Given significant reduction, ensure that enough CAS servers are deployed to handle failures and provide high availability

CAS Role – Memory

CAS memory is sized using a simple formula of 2GB + 2GB per-CPU core.

The per-core value assumes utilized CPU cores at peak (worst case failure), so this can get a little complicated Per−server CAS memory = 2GB + 2GB × total user count CAS server count in worst case × Mailbox mcycles per−user mcycles per−core × 0.25

Kemp Hardware Load Balancer & MS Exchange

Why Hardware Load Balancer…?

          Microsoft Recommends Service Aware, Pre-auth, SSO for Exchange & Lync Evenly distributes load(Http/Mapi{Exchange 2010}/Https) between CAS servers CA-FE now a stateless proxy server SSL Acceleration & Reverse Proxy Site HA & Geo Auto-failover “Kemp HLB” is Microsoft Certified & Few others too!!!

Engineered especially for Microsoft Exchange & Lync Lync / Application servers load balances too not limited to Exchange Play around with Trial Virtual Load Balancer from Kemp to experience the functionality and its features.

Active Directory

 Recommend deploying 1 AD GC core for every 8 Mailbox cores handling active load (assuming 64-bit GCs)  Size memory such that the entire NTDS.DIT can be contained within RAM for optimal query performance

Exchange 2013 Diagnostic & Perfmon

 Logs:  Microsoft\Exchange Server \V15\Logging\Diagnostics\DailyPerformanceLogs folder  Microsoft\Exchange Server\V15\Logging\Diagnostics\PerformanceLogsToBeProcessed  MS Exchange Health Management Service  Works towards assessing the health of Exchange components – OWA / EAS / EWS / ECP  It also captures information about activities such as the Calendar Repair Assistant, Mailbox Assistants, Mailbox Replication service, Migration service, OAB generation, and so on – V15\Logging root

Exchange 2013 Diagnostic & Perfmon

 Most of the logs generated in CSV which are helpful in troubleshooting issues – at times root cause.

 It fills the Exchange drive when not monitored – use data collector set to create resource policy and delete or follow

Removing Old Exchange 2013 Log Files

 Diagnostic logging – 70 New Added & 81 Old Removed  Removal of events for Public Folders and the addition of Front End Transport server events seem to be the biggest change. The ExchangeSA\OALGenerator one, commonly used to troubleshoot Offline Address Book issues, has been removed as well

Exchange 2013 - Sizing Tools

Sizing Tools – Exchange Role Calculator

     Role Requirements Calculator turns published sizing guidance into a modeling tool Try out various failure scenarios Understand the impact of different hardware & storage choices Provides scripts for DAG, database & copy creation Many new features  CAS sizing  Transport storage sizing  Multiple databases per-volume (JBOD) support  High availability architecture improvements http://aka.ms/E2013Calc

Sizing Tools – Exchange Role Calculator

Sizing Tools – Exchange Profile Analyzer

 Helps administrators collect estimated statistical information from a single mailbox store or across an Exchange Server organization     Can only be run on Exchange 2003(SP2) & 2007 Exchange View-Only Administrator permission to run Run during non-business hours to avoid performance issues(on CAS) if any Improve capacity planning method

Sizing Tools – Exchange Profile Analyzer

Sizing Tools – Collectlogs.vbs

 Calculate Exchange transaction logs generated for each hour in the day  A simple VBS script that will collect all files in a folder and output it to a log file  Use Task Scheduler to execute this script at certain intervals in the day (e.g. every 15 minutes)

Sizing Tools – EmailStatistics.ps1

 Script parses messaging tracking logs of your Exchange Servers and then tabulates the information into a CSV file  Demo Messages Received per Mailbox Per Day = Received Total Messages Sent per Mailbox Per Day = Sent Unique Total Average Message Size = Average of Received MB Total & Sent Unique MB Total

Sizing Tools – Processor Query Tool

 Quickly look up SPECint benchmark for a particular processor type  Takes average across multiple vendor submissions  Provides specific value to insert into Role Requirements Calculator http://aka.ms/ExProcQueryTool

Sizing Tools – Jetstress 2013

 Jetstress 2013 released March 2013  Validates that the physical deployment is capable of meeting specific performance requirements  Validates that the storage design is capable of meeting specific performance requirements  Finds weak components prior to deploying in production  Proves storage and I/O stability http://aka.ms/Jetstress2013

Sizing Tools – Jetstress 2013

 Jetstress is dumb tool that is used worldwide by thousands of Exchange professionals  When the Jetstress test reports a failure, Jetstress has not failed Windows Operating System

Windows Performance Counters Jetstress Application

Auto tuning Reporting and Verification Thread Dispatcher Background Log Checksummer Offline Log & Database Checksummer

Extensible Storage Engine (ESE)

Background Database Maintenance Transactional I/O Hardware

Storage Subsystem

Sizing Tools – Exchange Client Bandwidth Calculator

 This Calculator has been designed to profile the network bandwidth requirements of both Exchange on-premises and Exchange Online (Office 365) workloads.

 Hub site consolidation planning  Exchange service externalisation planning  Cloud requirements planning for Exchange Online (Office 365)  Demo

Exchange 2013 – Virtualization Sizing

 Sizing methods for Exchange 2013 for physical and virtual hardware is the same  However you need to consider the impact of a shared virtual infrastructure  ~10% for Hyper-V (follow vendors guidance for other hypervisors)  Co-locating Exchange with other virtual machines and workloads  Use resource reservation options to ensure that Exchange gets what it needs

Exchange 2013 – Virtualization Supportability



Supportability –

     Windows Server 2008 R2 Hyper-V Server 2008 R2 Windows Server 2012 Hyper-V Server 2012 Third-party hypervisors (SVVP)  Exchange Roles  Both Exchange 2013 roles supported for virtualization  Live migration & Host based clustering is supported

Exchange 2013 – Not Supported

 Dynamic Memory  Memory over commit / CPU over subscription  Hyper V snapshots  Differencing / delta disk  Apps on the parent OS

Exchange 2013 – Not Supported

Hyper-V replica

• • • Replica provides DR for “any” VM via log shipping to a remote hypervisor Use Data Availability (DAG) to obtain better HA & DR with Exchange 2010 & 2013 Replica makes sense for apps that don’t have DR capability built-in to the product Windows Azure Infrastructure Services, AWS or similar

Exchange 2013 – Dynamic Memory Impact

60 50 RPC Averaged Latency Available Bytes 40 30 20 10 0 6E+09 5E+09 4E+09 3E+09 2E+09 1E+09 0

Questions…?