Steve Tramack Sr. Engineering Manager Hewlett-Packard Corporation UNC307 Complete an evaluation on CommNet and enter to win!
Download ReportTranscript Steve Tramack Sr. Engineering Manager Hewlett-Packard Corporation UNC307 Complete an evaluation on CommNet and enter to win!
Steve Tramack Sr. Engineering Manager Hewlett-Packard Corporation UNC307 Complete an evaluation on CommNet and enter to win! Agenda Introduction to Sizing Exchange 2007 architectural changes – why? Server role sizing rules of thumb Impact of key hardware technologies Summary Why Sizing Matters Right-sized environment = happy users Exchange is the “canary in the coal mine” Server sizing is not like buying a PC Different applications demand different resources Changes from version to version, role to role “Sharing” resources isn’t easy Achieve ROI on overall investment Hardware is only a portion of the investment Reliability, SLA impact Key Sizing Elements Balanced hardware environment Make CPU the bottleneck Size for “peak average”, not “peak peak” Monday morning Repeatable Busiest servers Not “average average”, either Know when to use percentiles Average active connections: 1441 Sizing Methodology Know thyself • What is your definition of a “User” • How do your users behave? Establish baselines • Apply sizing best practices • Utilize tools, resources • Hardware, SLAs, thresholds Test, test and re-test • Repeatable results Create a balanced system • Invest wisely • Address bottlenecks Deploy and proactively monitor Agenda Introduction to sizing Exchange 2007 architectural changes – why? Server role sizing rules of thumb Impact of key hardware technologies Summary Exchange Server 2003 “Pain Points” Scalability limitations Store size tied to SLA 32-bit = 4GB memory space VM fragmentation Max 1.2GB Database Cache I/O pressures Kernel, user mode limitations Sizing considerations I/O, I/O, it’s off to size I/O Larger disks provide unbalanced solution Example heavy user with large mailbox and Blackberry could exceed 3 IOPs Memory Limitations Each 32-bit process: 4GB VM /3GB for large Exchange servers Reduce VM fragmentation Larger ESE cache User …which creates Kernel pressure /USERVA:3030 for PTEs Security Tokens for active sessions drain paged pool 250MB on a server with /3GB 8K token size = ~3,700 users See hotfix 912480, KB912376 PCI-Express drives nonpaged pool consumption Kernel Exchange Server 2007 Architecture (1 of 2) Sizing Perspective x64 everywhere 40-bit physical (1TB) / 48-bit virtual (256TB) address space Windows support 43-bit addressing – 16TB Dramatically changes the baseline I/O footprint Updated search engine (30x improvement) New server roles / increased demands placed on existing roles LCR / CCR RTM version impacted I/O on CCR passive node Addressed by SP1 ESE cache persisted 40% less I/O on passive node than active Exchange Server 2007 Architecture (2 of 2) Sizing Perspective I/O profile changes ESECache from 1.2GB 25 GB (32GB server) Common items cached in memory More aggressive read-ahead caching 4K 8K database page size More SGs / DBs (up to 50) 20MB checkpoint depth per SG More SGs = more checkpoint depth per user Less physical writes Online defrag write I/O reductions I/O coalescing increased 64K 1MB SP1 removes page dependencies Improvements in I/O, reduction in memory and CPU Database Cache Comparison x64 is Good! More cache = Less I/O Exchange Server 2003 test (4GB) Database cache tuned to max value (1.2GB) Exchange Server 2007 test (32GB) Database cache grew to roughly 25GB; stabilized at 22GB “Dip” in first hour related to LoadGen behavior 64-bit Impact (Mailbox Server) 1 0.8 IOPS I/O reductions based on increasing database cache per user More reads from memory result in fewer disk transfers 0.6 Reads/sec/user 0.4 Writes/sec/user IOPs/user 0.2 0 0.25 1 2.5 4.5 6 RAM/user (MB) 1 0.8 IOPs RAM/user remained constant Spreading users across more SGs means more checkpoint depth Reduces writes/sec 1GB mailbox size 20KB checkpoint/user Ultra-heavy user profile 0.6 Reads/sec/user 0.4 Writes/sec/user 0.2 IOPs/user 0 20 40 80 Checkpoint Depth / user (KB) 140 1 GB mailbox size 6MB RAM/user Ultra-heavy user profile Hardware: ProLiant DL585 4 x dual core AMD 2.2 GHz User profile: 4000 Outlook 2003 online users simulated with LoadGen, 100MB mailbox size, 17 local deliveries/sec Agenda Introduction to sizing Exchange 2007 architectural changes – why? Server role sizing rules of thumb Impact of key hardware technologies Summary Exchange Server 2007 Roles Enterprise network Other SMTP Servers I N T E R N E T Hub Transport Services Edge Services Routing Hygiene Routing OWA ActiveSync, POP, IMAP, RPC / HTTP … Programmability: Web services, Web parts Client Access Services Policy Unified Messaging Services Applications: Protocols: PBX or VoIP Voice Messaging Mailbox Services Mailbox Public Folders Fax Mailbox Sizing What load factors influence sizing? Mailbox size and count User profiles User type Send/rcv per day (50K Continuous messages) Replication Light 5 sent / 20 received Cache Warming Medium 10 sent / 40 Transactional received I/O configuration Heavy 20 sent / 80 for MDBs and logs received Very heavy 30 sent / 120 Connections received Mobile devices Online mode versus cached mode Database Estimated cache / user IOPS/user 2 MB 0.11 3.5 MB 0.18 5 MB 0.32 5 MB 0.48 Mailbox Sizing Rules of thumb CPU Up to 12 cores 500 – 1,000 mailboxes / core (based on user type, 75% concurrency) Memory Depends on user types, storage groups, other factors 32GB max (cost effective) Network Min 1 x Gbit; use multi-function NICs with iSCSI With CCR, dual Gbit NICs. Consider NIC teaming and separating traffic Storage Group Count Exchange 2007 Service Pack 1 1-4 2GB 5-8 4GB 9-12 5GB 13-16 6GB 17-20 7GB 21-24 8GB 25-28 9GB 29-32 10GB 33-36 11GB 37-40 12GB 41-44 13GB 45-48 14GB 49-50 15GB Disk Balance I/O and Capacity Content Indexing +5% capacity into database LUN size Overhead / white space Amount of mail sent/received per day Factors in message size SLA factored into sizing Mailbox Size = Mailbox Quota + White Space + (Weekly Incoming Mail x # dumpster weeks) - Consider 20% overhead for database, log LUNs Log Size = (Logs/SG x 1MB) + Move mailbox size + incremental restore size Max database size Recovery Storage Group Backup to Disk Deleted items retention (+15 – 30% for default 14 days) Effect of Large Mailboxes Stated profile IOPs based on cache mode 250MB online mode, OWA clients: 1.5x database reads As mailbox size doubles, read IOPs double >20,000 items per folder impacts IOPs Watch restricted view requests in critical path folders Adding columns to a view, sorting on new columns, finds and searches Key: disk latencies <20ms 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 250MB 500MB 1 GB 2 GB DB Read IOPs/Mailbox 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 5MB cache 10MB cache 15MB cache DB Read IOPs/Mailbox 20MB cache “Cache is King” Impact of cache warming 8C mailbox server with 32GB RAM. 15K MMB4 users (LoadGen). Exchange Server 2007 RTM code Cache Size (MB) I/O issues reflected in RPC Average Latency Exchange 2003: avg. 20-25ms Exchange 2007: avg. <10ms User doesn’t typically perceive impact IOPS/user Plan maintenance windows Impact to users? Why not Scale Up Further? Connection limitations to STORE process Memory 32GB @ 5MB/user = 6000 users 8 cores typically sufficient Watch other load generating factors Scaling up increases risk for DR Storage still the dominant cost… MSIT workload is only .33 IOPs Lower cost through managing IOPS versus consolidating mailbox servers 4 - 16 core test 70 60 57% 50 40 33% 30 22% 20 16% 10 0 4 cores 8 cores CPU utilization RPC lat avg 12 cores 16 cores RPC latency max HP ProLiant DL580 G5, 4,000 LoadGen heavy users, no A/V or CCR, 8-hour simulation day, 56GB RAM 1GB mailboxes, 30 MDBs, EVA8000 with 58 DB disks, 8 Logs. Exchange 2007 SP1 Reference Configuration “What’s a ‘user’? What does this site look like?” 12,000 users – 3 x CCR clusters Servers (each CCR Source and Target) 4,000 “users” per CCR cluster 8 cores (2 x 4 or 4 x 2) with Intel E5504 2.0Hz 80W processors 24GB RAM (6 x 4GB RDIMMs) Separate NIC ports for production and cluster 2 local disks (O/S, binaries, page file) in a RAID1 set Storage (each CCR Source and Target) 21 SGs/server (1 MDB/SG) 190 – 191 users / MDB 36 x 300 GB 10K SFF SAS disks (RAID5) for databases 9 x 300 GB 10K SFF SAS disks (RAID10) for logs Databases and logs include disk sparing 5 x 300GB 10K SFF SAS disks (RAID5) for restore LUN CAS Sizing What load factors influence sizing? Message conversion by non-MAPI clients OWA rendering Unified namespace Services Internet protocol handling (IMAP, POP3) Outlook Anywhere Airsync Autodiscover, Availability OAB distribution 1 CAS and 2 MBX servers. 2C 2.2GHz ProLiant DL385 servers. 4GB RAM. 100% OWA concurrency Outlook Anywhere, Airsync and CAS Outlook Anywhere Additional scalability with Windows Server 2008 and multiple IP addresses 6 to 8 TCP connections for baseline Outlook configuration Memory concerns 61K / virtual connection Min 4GB for each IP Address http://technet.microsoft.com/enus/library/cc540453.aspx Airsync Outlook 2003 TCP to CAS Outlook Anywhere (outside firewall) Online Mode 6 Cached Mode 6 Outlook 2007 TCP to CAS Outlook Anywhere (outside firewall) 12,000 direct push users/devices Online Mode 6 Based on heavy workload of 80 msgs/day Cached Mode 8 Rules of Thumb CPU Memory Min Rec Max Ratio CAS 1 core 4 cores 6 cores 4 : 1 per AD site CAS + Hub 1 core 4 cores 6 cores 4:1 Min Rec Max CAS 2GB 2GB/core 16GB CAS + Hub 8GB 2GB/core 16GB Deploy CLR QFE for Memory Disk Minimal requirement Network Profile-dependent 1 x Gbit recommended CAS example configuration 12,000 users –3 x CCR clusters 3 Servers, each with: 4 cores (1 x 4) 12 GB RAM 1 x Gbit NIC 2 x 146 GB disks (O/S, binaries, page file) in a RAID1 set Could deploy Hub + CAS combination (ex. branch offices) Hub Transport Server What load factors influence sizing? Mail queue stored in JET database Recommend increasing database cache to 512MB CCR (Transport Dumpster) Message rate Hub Transport server database I/O (~21 Message tracking msg/sec) Average size Total IOPS per message Number of enabled (approximately 46 KB avg message size) transport agents Log write I/Os per message (sequential) Antivirus Database write I/Os per configuration message (random) Third-party Database read I/Os per message (random) applications SP1 Default Cache (128MB) 512MB Cache 26.71 11.67 3.44 3.28 11.69 7.13 10.48 0.42 Based on 12SG test with 18MB Max Dumpster size per SG CPU Memory Min Rec Max Ratio No A/V 1 core 4 cores 12 cores 7:1 w/ A/V 1 core 8 cores 12 cores 5:1 Min Rec Max Normal 2GB 1GB/core 16GB Large queue Varies 2GB/core 16GB Large queues: 3KB/msg, 1KB/recipient EdgeSync: 4KB/mail enabled object Deploy CLR QFE for Memory Disk Separate mail.que and logs from system disk (RAID 10) Transport dumpster capacity Network Gbit recommended Hub example configuration 12,000 users – 3 x CCR clusters 2 Servers, each with: 8 cores (2 x 4) 8GB RAM (increase if experiencing queuing) 1 x Gbit NIC Local hardware RAID controller with BBWC 2 x 146 GB disks (O/S, binaries, page file, transaction logs) in a RAID1 set 6 x 146 GB disks (DB, protocol & message tracking, AV quarantine, transport dumpster) in a RAID10 set Edge Transport Sizing What load factors influence sizing? Similar to Hub Message rate Average message size Number of enabled transport agents Antivirus scanning Antispam Performance metric Value SMTP Connections/Sec 55 % Connections Accepted 80% SMTP Messages IMF Scanned/Sec 3.7 % SMTP Messages passed IMF Scanning 80% SMTP Messages A/V Scanned/Sec 3 Avg. Message Size 70 KB SMTP Connection CPU Utilization on Edge Server 20% ** analysis How much of a target are you for spam / virus attacks? ** 2-socket, dual-core AMD Opteron 275 2.2 Ghz processor. Microsoft example numbers Edge Sizing Rules of thumb CPU Edge Memory Min Rec Max Ratio 1 core 2 cores 6 cores 5:1 Normal Min Rec Max 2GB 1GB/core 16GB Deploy CLR QFE for Memory Disk Network Increase database cache to 512KB in EdgeTransport config file Edge Transport database I/O Edge I/O Log writes I/Os per message ~ 11 Database write I/Os per message ~ 4.5 Database read I/Os per message ~ 2.5 Gbit recommended Edge example configuration 12,000 users – 3 x CCR clusters 2 Servers, each with: 8 cores (2 x 4) 8GB RAM 1 x Gbit NIC Local hardware RAID controller with BBWC 2 x 146 GB disks (O/S, binaries, page file, transaction logs) in a RAID1 set 6 x 146 GB disks (DB, protocol & message tracking, AV quarantine) in a RAID10 set UM Sizing What load factors influence sizing? Outlook Voice Access Voice mail storage codec WMA PCM GSM 06.10 Call answering and forwarding Concurrent calls Concurrent fax messages UM performance testing 4 x T1 with 23 channels each = 92 channels or calls 7 – 30 sec msgs Single CPU testing – calls missed instead of voice mail 2 CPU 4 CPU Average CPU 93 83 45 Current Calls 78 85 86 Delayed Calls 597 12 0 Max VM Size (KB) 23 23 25 Average Call (sec) 59 58 55 User Response Time (ms) 550 224 0 700 120 600 100 500 80 4 CPU 60 2 CPU 40 1 CPU 20 0 1 CPU 1 CPU 400 2 CPU 300 4 CPU 200 100 CPU % during load testing 0 Delayed calls during load testing Rules of thumb CPU UM Memory Min Rec Max Ratio 1 core 4 cores 6 cores Varies Concurrency Use Erlang model Codec Calls/sec WMA 60 GSM 75 Disk is also codec-dependent Equates to 2K – 10K users / server Normal Min Rec Max 2GB 1GB/core 4GB Network Gbit recommended UM Example Configuration 12,000 users – 3 x CCR clusters 2 Servers, each with: 8 cores (2 x 4) 8GB RAM 1 x Gbit NIC 2 x 146 GB disks (O/S, binaries, page file) in a RAID1 set AD / GC Sizing No change with Windows Server 2008 64-bit architecture leveraged by AD/GC 32-bit DC/GC: 1 core per 4 mailbox cores 64-bit DC/GC: 1 core per 8 mailbox cores Must have enough memory to cache entire AD database (NTDS.DIT) in memory to follow these ratios Allows for performance boost without I/O impact 1 GC near 10K (32-bit) to 20K (64-bit) clients No dedicated AD site for Exchange Message routing maps to AD topology 12,000 users – 3 x CCR clusters 2 Servers, each with: 2 cores (1 x 2) 4GB RAM (depends on size of .DIT) 2 x Gbit NIC teamed 2 local disks (O/S, binaries, page file NTDS.DIT, logs) in a RAID1 set Agenda Introduction to sizing Exchange 2007 architectural changes – why? Server role sizing rules of thumb Impact of key hardware technologies Summary CPU Considerations Megacycles/U = # cores * MHz * CPU % / # users Biggest single influencing factor: Front-Side Bus 41% reduction in Mcycles/U when increasing FSB from 1066 to 1333MHz Sweet spot: 2.0GHz QC “Clovertown” processor 2-socket Intel CPU Comparison $1,800 $1,600 $1,400 $1,200 $1,000 $800 $600 1.00 1066 MHz 0.81 0.71 0.90 0.80 0.74 1333 MHz 0.44 0.6 0.51 0.6 0.70 0.60 0.50 Price 0.40 CPU% 0.30 Mcycles/U $400 0.20 $200 0.10 $0 0.00 3.00Ghz DC 3.33Ghz DC 1.86Ghz QC 2.00Ghz QC 2.33Ghz QC 3.00Ghz QC 3.00Ghz QC 5160 X5260 E5320 E5335 E5345 X5365 X5450 HP BladeSystem BL480c, 4,000 LoadGen heavy users, 8-hour simulation day, 22GB RAM (5MB/user) , 1GB mailboxes, 30 MDBs, EVA8000 with 58 DB disks, 8 Logs. Exchange 2007 SP1 CPU Considerations Intel “Nehalem” (45nm µArchitecture) processors Hyper-threading Turbo Boost mode New shared memory architecture (QuickPath) helps alleviate front-side bus bottleneck Higher bandwidth, lower latency 118% performance improvement at 32% price delta 70% 60% $1,800 64% $1,600 55% $1,400 50% $1,200 40% $1,000 31% $800 30% $600 20% $400 10% 2.2 1.9 .87 0% $200 $- E5405 2.0GHz 2P/8C X5450 3.0GHz 1P/4C CPU % Mcycle/U X5560 2.8GHz 1P/4C (HT) Cost HP ProLiant DL380 G6 and BladeSystem BL460c , 4,000 LoadGen heavy users, 4-hour simulation day, 22GB RAM (5MB/user), 1GB mailboxes, 30 MDBs, EVA8000 with 58 DB disks, 8 Logs. Exchange 2007 SP1 Memory Considerations No longer “4GB and forget about it” Memory types Power and Thermals $6,000 Advanced memory buffer on FB-DIMM draws ~5 watts Low power memory option save 1KW/rack Memory DIMM size $5,000 $4,000 Cost, speed considerations DIMM slots and population rules 1GB UDIMM $3,000 2GB UDIMM 2GB RDIMM UDIMM max = 24GB 4GB RDIMM $2,000 RDIMM Price / RDIMM Price / GB Cost of 32GB 2GB $105 $52.50 $1,680 4GB (QR) $240 $60.00 $1,920 8GB $990 $123.75 $4,060 $1,000 $0 8GB RDIMM Disk Considerations (1 of 2) Storage requirements are a multi-dimensional problem Balance IOPs (trans and non-trans), capacity and backup Read/write latency <20 ms, with spikes no more than 50ms SATA considerations Enterprise-class / midline SATA disks advancements More resilient to rotational vibration Enterprise-class duty cycles Requires battery-backed controller cache Sweet spots based on: Disk technology, RAID level, mailbox size and IOPs Consider power and cooling OpEx costs Host-based IOPs/spindle Exchange workload RAID 10 RAID 5 15K SFF SAS 180 – 230 150 15K LFF SAS 165 135 15K FC / SCSI 150 120 10K SFF SAS 130 95 10K LFF SAS 115 90 10K FC / SCSI 100 80 7.2K midline SAS / SATA 45 35 RAID 6 110 80 30 Disk Considerations (2 of 2) Other factors can impact IOPs/spindle and MB/s Controller bottlenecks Fill factor percentages Allocation Unit size IOPs based on fill factor 10200 10100 10000 9900 9800 9700 9600 9500 63% filled Allocation Unit Size 1000 800 864 714 600 400 200 0 Default 64K IOPs 80% filled Controller bottleneck 160 158 156 154 152 150 148 146 144 142 140 10200 20 18 16 14 12 10 8 6 4 2 0 10100 10000 9900 9800 9700 9600 9500 9400 MB/sec IOPs/spindle 9300 70 disk, 5.7 TB Based on MDS600 (single and dual) testing with Smart Array P700M controller, single server, 300GB LFF 15K SAS disks 70 disk, 7.2 TB 100 disk, 8.7 TB IOPS Read (ms) Write (ms) RAID Level Impact Array Rebuild Times Under Load RAID 10 and 5 are both viable options for Exchange Server 2007 mailbox server storage Reduced IOPS + increased capacity + Exchange Server 2003-optimized storage (smaller, fast disks) make RAID 5 more compelling RAID 10 rebuild impacts single disk; RAID 5 rebuilds parity across all disks in the RAID set SAS benefits over SCSI Array Rebuild Times RAID 10 RAID 5 RAID 6 10K SFF SAS 130 95 80 % reduced 13% 19% 30% 7.2K MDL SAS / SATA 45 % reduced 16% 35 30 Disk Technology Array Rebuild Impact on IOPs/spindle SAS RAID 10 U320 RAID 10 SAS RAID 5 U320 RAID 5 36% TBD 0 1 2 3 4 5 6 7 8 Time (hours) ProLiant DL385 2 Dual-Core CPU (2.2GHz), 4GB RAM, 1500 MMB3 users, 100MB mailboxes, SAS 16 DB disks, 4 Logs. Search/Indexing=OFF. Exchange Server 2007 beta. Rebuild priority = low 9 10 Storage Fabric Considerations I/O reductions mean increased options for enterprise storage deployment Local array controllers with SAS / SATA connectivity Not JBOD – server-based array controllers with BBWC App-centric storage Fibre Channel and storage virtualization Storage as a data center service to many apps iSCSI Multi-function NICs provide performance enhancements, CPU relief, iSCSI booting Microsoft’s initiator integrated with MPIO, allowing for dual path for high availability and load balancing Virtualization impact Use same basic sizing rules of thumb for Exchange 2007 SP1 on Hyper-V Scales well from 1 to 4 processors (per VM) Comparable overall CPU burn, IOPs/user Storage and network design require consideration of CPU impact to the Root All guest I/O requests are serviced by the Root More pieces to monitor Performance in the Root Performance in the Exchange VM Sizing Resources Benchmarks? ESRP Sizers Microsoft’s Exchange Server 2007 Mailbox Server Role Storage Requirements Calculator HP Sizing Tool DiY LoadGen JetStress Blogs, TechNet White papers Agenda Introduction to sizing Exchange 2007 architectural changes – why? Server role sizing rules of thumb Impact of key hardware technologies Summary Summary - update Move to x64 was necessary x64 drives new recommendations More memory for ESEcache = reduced I/O footprint Different storage planning paradigm Continuous replication provides higher Disk, fabric options still driven by balancing capacity, I/O, recoverability (SLA) demands Network plays bigger role New roles, technology to consider Sizing “process” remains the same Still need to understand “user” Baseline and test Balance environment Apply best practices and proactively monitor Resources www.microsoft.com/teched www.microsoft.com/learning Sessions On-Demand & Community Microsoft Certification & Training Resources http://microsoft.com/technet http://microsoft.com/msdn Resources for IT Professionals Resources for Developers www.microsoft.com/learning Microsoft Certification and Training Resources Call to Action Learn More! Related Content at TechEd on “Related Content” Slide Attend in-person or consume post-event at TechEd Online Check out online learning/training resources http://technet.microsoft.com/exchange/2010 http://technet.microsoft.com/office/ocs Try It Out! Download the Exchange Server 2010 Beta Evaluation http://www.microsoft.com/exchange/2010/try-it Get a 5-Day Trial of Office Communications Server 2007 R2 https://r2.uctrial.com/ © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.