Veeam Backup & Replication: Tips and Tricks

Anton Gostev

Veeam Software

@Gostev Doug Hazelman

Veeam Software

@VMDoug #vmworldsponsor SPO3292

Quick Overview of v6 Architecture

    Backup servers Backup proxy servers Backup repositories “Automated everything” Intelligent load balancing Centralized management via Enterprise Manager

Scaling your backups

3 simple rules

Scaling your backups

1.

Keep the management server happy ● ● ● Disable default proxy (requires 6.1 or later) Allocate enough RAM for job manager processes Keep concurrent (running) jobs under 100 per management server 2.

Backup proxy servers: the more, the better?

● ● ● No! Too much load on storage and network Use max concurrent tasks limit on proxies or repositories Watch for “job timed out waiting for resources” messages 3.

Backup repository considerations ● ● ● Be careful with the reversed incremental backup mode!

Limit concurrent jobs on backup repositories to a reasonable amount Use ingest rate throttling for cross-SAN backups

A word on backup repositories

 Don’t underestimate the importance of performance!

● By far, the most commonly reported bottleneck  What makes the best backup repositories?

● ● ● Windows or Linux server (can be same as backup proxy server) Local storage, DAS or SAN mounted for physical server pRDM disk (vSphere 5+), or iSCSI LUN connected via in-guest iSCSI for virtual server  Sub-optimal backup storage ● ● NAS or network share VMDK on VMFS (size and recoverability considerations)  RAID level ● If you can afford it, use RAID10 (again, performance)

Processing Modes

All you need to know

Direct SAN Access: The good

  Fastest processing mode Least impact on production ● ● Backup processing is fully offloaded to dedicated backup proxy servers Backup traffic is isolated to the storage network (aka LAN-free)  Does not impact consolidation ratio, so cheapest too

Direct SAN Access: The bad

 Supports block storage only ● ● FC (fibre channel): physical backup proxy server only!

iSCSI: physical and virtual backup proxy servers both supported   Might be hard for beginners to setup ● See Veeam Forums FAQ for step-by-step guide  Physical backup proxy server requirement for FC SAN ● ● Might not go along well with your virtualization project Consider repurposing older servers Manual datastore mapping might be required ● For certain SANs, B&R may not able to detect proxy connectivity

Direct SAN Access: The ugly

  What’s the worst that could happen?

Windows re-signaturing your VMFS LUNs!

● ● vSphere will no longer recognize datastores Don’t panic, VMware Support should be able to fix  Three easy ways to get into trouble 1.

2.

3.

Windows Explorer automounting new volumes (but not with Veeam) Clicking Disk Management snap-in popup without reading Giving Local Administrator rights to random people

Direct SAN Access: The safe way

 Present VMFS LUNs to backup proxy server as read-only ● Most SANs support it these days —chase your vendor if yours does not  Disable automount on your backup proxy servers ● ● Do it the right way: use

SANPolicy

Windows setting!

Veeam backup proxy server setup does this automatically for you   Disable Disk Management snap-in with Group Policy

User Configuration > Administrative Templates Window Components > Microsoft Management Console > Restricted/Permitted snap-ins > Disk Management

Keep Local Administrator rights on backup proxy servers to yourself ● ● Cannot really do this for default proxy due to FLR requirement Another reason to use dedicated backup proxy server!

Direct SAN Access: Tips & tricks

 Got a fast SAN? Get a modern backup proxy server!

● Multi-core CPU (compression) and fast RAM (inline deduplication)    Update firmware and drivers across the board Disabling MPIO may increase performance iSCSI SAN? Tweak TCP/IP on backup proxy

netsh interface tcp set global autotuning level = disable

 Increase read-ahead buffer ● ● Default is 4MB (optimal setting for most SANs) To change, create the new value in bytes:

VddkPreReadBufferSize (DWORD)

Hot Add: The good

 Easy to setup —very little planning involved ● Any Windows VM can be made a Hot Add backup proxy  Fast data transfers with any storage ● Direct storage access (albeit through ESXi storage stack)  Supports all types of storage (including NFS) ● ● Shared storage: at least 1 backup proxy server per vSphere cluster Local storage or DAS: at least 1 backup proxy server per host  Use your existing Windows VMs (save on licensing) ● ● Data processing engine process runs with lower priority (6.1) Further CPU usage reduction in 6.5

 Allows for 100% virtual deployment

Hot Add: The bad

  Not as mature as other modes Affects your consolidation ratio ● ● Backup proxy servers take host resources Ultimately means more ESXi hosts, and more VMware licenses  Hot Add process itself is slooow ● Can take up to 1 –2 minutes to complete for each VM—adds up quickly!

 Hot Add as a vSphere feature has a number of limitations ● ● Good news —many are being removed as VDDK matures See FAQ on Veeam forums FAQ for the complete list

Hot Add: The ugly

 Snapshot removal problems due to locks ● Veeam B&R: multiple hooks in place to work around  CBT must be disabled on backup proxy VM ● Prevents stun on Hot Add due to CBT initialization  NFS-specific issue ● Extended VM stun on hot remove in some scenarios

Hot Add: Tips & tricks

 Add extra virtual SCSI controller to backup proxy server ● ● A single SCSI controller can have a maximum of 16 disks attached Concurrent jobs on the same backup proxy server can result in more!

  Try increasing read-ahead buffer ● Seems to really help with certain NFS storage  Keep vSphere and Veeam up to date ● ● Single block size in VMFS5 removes the most common hot add issue Latest Veeam Backup & Replication will have latest and greatest VDDK version Avoid cloning backup proxy VM ● For example, to provision additional backup proxies

Network (NBD): The good

   Easy to setup —in fact, no setup is required ● Any existing server (physical or virtual) would do  Supports all types of storage, including NFS ● Server placement does not matter (unlike with Hot Add) Very quick to initialize data transfer Can be quite fast —with 10Gb Ethernet

Network (NBD): The ugly

 Painfully slow performance on 1Gb Ethernet ● Average speed reported is 10-20 MB/s  Leverages ESXi management interface

Network (NBD): Tips & tricks

1 Gb Ethernet  Use for sites with low change rate ● Works faster than other processing modes in such conditions  Keep at least one Hot Add backup proxy server around ● Full VM and virtual disk restores take

forever

over NBD  Keep in mind intelligent load balancing algorithms ● Network backup proxy servers have lowest priority!

 Upgrade to 6.1 or later ● Improved network proxy location awareness

One last thing

 

This hack significantly reduces supportability!

Cut up to 5 minutes of processing time

per VM

by disabling VDDK logging  Apply in stable environments only!

● Create the new value and set to 1:

DisableVDDKNetworkOutput (DWORD)

Deduplicating Storage

Yes, you can afford it!

Deduplicating storage: The good

 What gives? Global dedupe!

● ● Deduplication across backup files from different jobs Perfect for long-term backup archival  Top hardware appliances among Veeam users ● ● ● EMC DataDomain ExaGrid HP StoreOnce  Top software appliances among Veeam users ● ● ZFS-based appliances StarWind  Windows Server 2012 dedupe is

awesome

Deduplicating storage: The bad

 Hardware appliances are expensive ● Although they do provide

excellent

dedupe ratio  Software appliances are resource hogs ● Both performance and dedupe ratio are sub-par, too  Windows Server 2012 dedupe is ● ●

awesome

Included free of charge —start using it today!

Provides very decent dedupe ratio

Deduplicating storage: The ugly

   Random access performance is lacking ● ● A problem for

all

solutions featuring inline deduplication Typically insufficient out-of-the-box for large-scale vPower usage Exception: post-process deduplication ExaGrid ● ● Raw disk landing zone (full-speed vPower from recent backups) Veeam-specific logic further optimizes performance  Windows Server 2012 ● ● ● Backups “land” on raw storage at full speed Only old backup files are deduplicated —great for vPower Decent speed even off already deduped backups

Deduplicating storage: Tips & tricks

 Already own storage with inline deduplication?

● Inline data “rehydration” process is what makes vPower slow  Reduce the block size in Advanced job settings ● ● WAN (256KB) and LAN (512KB) Reduced block size might impact backup performance  Use Linux-based backup repository ● Large client cache, or even caching client file system (FS-cache), can

significantly

improve vPower performance

Deduplicating storage: Tips & tricks

(continued)    Keep Veeam dedupe on Use incremental backup mode ● Choice of synthetic or active fulls depending on actual storage For best dedupe ratio on device side… ● Disable compression (significantly increases amount of data transferred from backup proxy server to backup repository over network)   For best backup performance and smallest window ● Keep compression at default If you like to avoid extremes… ● Set compression to Low (dedupe-friendly)

Deduplicating storage: Tips & tricks

(continued) Got more than one deduplicating storage device?

 Use internal replication to sync backups offsite!

● ● Extremely traffic-efficient approach Many customers use and report great success!

 Keep the backups imported for easy DR ● To automate repository refresh in DR site, use:

Get-VBRBackupRepository -Name "DR_Repository" | Sync-VBRBackupRepository

WAN Accelerators

Your WAN on steroids

WAN accelerators: The good

 Two types of WAN accelerators ●

Caching WAN accelerators

provide significant bandwidth savings with Veeam replication, but are typically quite expensive ●

Transport layer WAN optimizers

are unlikely to offer significant bandwidth savings with Veeam replication, and are usually cheap  Both improve reliability of TCP ● ● ● Long distance wireless or satellite links IPsec rekey operations on a VPN tunnel WAN links with high jitter, packet loss or occasional drops   Both allow long-running jobs to finish more consistently ● For example, initial replication over network Both improve WAN utilization for most workloads

WAN accelerators: What’s hot?

 Top caching WAN accelerators among Veeam users ● ● ● Cisco WAAS Riverbed SilverPeak  Top WAN optimizers among Veeam users ● Hyper-IP

WAN optimizers: Tips & tricks

 Veeam Backup & Replication leaves little room for bandwidth reduction by WAN optimizers   Built-in WAN optimizations in v6: ● ● Multiple TCP streams to maximize throughput Network traffic compression Consider using on unreliable networks, but don’t expect them to add you extra bandwidth ● Update to Backup & Replication 6.1 Patch 1 before evaluating  Veeam Backup & Replication not using all available bandwidth?

● Increase the amount of TCP streams (default is 5)

DownloadStreamsNumber (DWORD)

Caching WAN accelerators: Tips & tricks

 Multiple TCP streams can cause issues!

● Disable multiple streams in B&R traffic throttling settings  Disable network traffic compression in Veeam ● Low (dedupe-friendly) compression level might be a better option  If required, have network admins configure bypass on Veeam backup proxy servers to avoid polluting caches

Veeam Backup & Replication 6.5

We never stand still

What’s coming in 6.5

    Veeam Explorer for Microsoft Exchange Veeam Explorer for SAN Snapshots VMware vSphere 5.1 support Windows Server 2012 support

Veeam Explorer for Microsoft Exchange

   Visibility into Exchange VM backups ● Immediate: No need to provision storage, restore the VM or restore the mailbox store ● ● Agentless Requires no special backups or metadata collection —even works with existing Veeam backups (and SAN snapshots) Free!

● Included in all versions of Veeam Backup & Replication 6.5, including Free Edition ● Eliminates need for expensive standalone tools licensed per-mailbox Currently available in “exclusive beta”

Just restored a 145 GB #MSExchange Public Folder database in 2 min. using @veeam . Then restored a single item all under 10 min. SWEET!

The new #veeam explorer for exchange looks veeamy. That's right, I just made up a new word (it means awesomesauce) Even the *beta* of @veeam Exchange Explorer works a treat. Saved literally, hours of work.. and saved my bacon. #recommend Really excited with the new @veeam Explorer beta for #Microsoft Exchange VM backups - we have lots of interested customers ready for this!

Veeam Explorer for Microsoft Exchange

(continued)   Capabilities ● ● ● Browse: familiar Explorer-type interface Search: familiar Outlook-like Find, including Advanced Find Export: export to PST file, MSG file or attachment Uses cases ● ● ● E-discovery Item-level restore: export and send to affected user Mailbox archive  Supports Exchange Server 2010

Veeam Explorer for SAN Snapshots

   Veeam restores from SAN snapshots Supports tiered data protection strategy Perform all restores through familiar, easy-to-use Veeam interface  Supports HP StoreVirtual VSA and HP LeftHand

SAN snapshots + Veeam restore = Best RPOs and RTOs for operational recovery

Veeam Explorer for SAN Snapshots

(continued)   Fast: recover entire VM or individual items in < 2 minutes ● ● Fully automated: clone & promote snapshot, present to vSphere, clean up Restores directly from VM files on the SAN snapshot: no staging or intermediate restores required  Flexible ● ● ● Specific VM Individual guest files: Windows, Linux, et al Individual Microsoft Exchange items Free ● Worry-free: automated process eliminates human errors and protects integrity of SAN snapshots and production LUNS ● Agent-free: no agents to deploy on hosts or VMs ● Literally free: included in all editions of Veeam Backup & Replication 6.5, including Free Edition

Questions? Comments?

Thanks for attending!

FILL OUT A SURVEY

AT WWW.VMWORLD.COM/MOBILE COMPLETE THE SURVEY WITHIN ONE HOUR AFTER EACH SESSION AND YOU WILL BE ENTERED INTO A DRAW FOR A GIFT FROM THE VMWARE COMPANY STORE

Veeam Backup & Replication: Tips and Tricks

Anton Gostev

Veeam Software

@Gostev Doug Hazelman

Veeam Software

@VMDoug #vmworldsponsor SPO3292