Disaster Recovery

(Business Continuity Planning) Tim Babco

AGENDA

• What is BCP?

• Key BCP Components • How to Implement?

• How to Maintain?

• Effort Required • Value Obtained • Q&A

Who is Poolcorp?

• • • • • •

World’s largest distributor of swimming pool supplies, equipment and related leisure products ~$2 billion in revenues >3,600 employees; 285 locations; 8 countries >100,000 products >70,000 customers Headquartered in Covington, Louisiana Headquarters in “Hurricane Alley”

What is BCP?

•

Task of identifying, developing, acquiring, documenting , and testing procedures and resources that will ensure continuity of a firm's key operations in the event of an accident , disaster , emergency , and/or threat . It involves: (1) Risk mitigation planning (reducing possibility of the occurrence of adverse events) (2) Business recovery planning (ensuring continued operation in the aftermath of a disaster) Keep the business running successfully

What is BCP?

• Business Continuity Planning (BCP) and Disaster Recovery (DR) are often used synonymously • Continuum: Enterprise  • Corporate functions • Remote locations • Succession Planning • Prevent and Recover Individual

Keep the business running successfully

Key BCP Components

• Needs assessment • Employees • Communications • IT Infrastructure • Recovery site logistics • Third party information • Supplies • Pre-event protection steps • Trigger points • Municipal, state, federal interaction and updates

Many important facets

Needs Assessment

• RTO – Recovery Time Objective • Amount of down time for each critical function before outage threatens company survival • RPO – Recovery Point Objective • How old can the data be before it is so out of date that recreation is not practical or possible • Consider Time of Year, Month, etc.

• Economic Benefit • Cost of protection vs. cost of down time • Hard costs and opportunity costs

Clearly define what’s important

Employees

• Current and complete contact information

Know how to reach employees

Employees

• Current and complete contact information

Employees know key BCP contacts

Employees

• Personal BCP plan • Home • Belongings • Immediate family members • Extended family members • Pets, livestock

Employees can’t be productive if worried about personal items

Employees

• Roles • Executive team • Make decisions • Delegations of authority • Communications team • Internal • External

Execute quickly and correctly

Employees

• Roles • Core team • Coordinate detailed plan execution • Tiered response teams • Tier 1 – IT only; sent when disaster impact predicted • Tier 2 – Employees with critical functions; sent when disaster impact is imminent • Tier 3 – Important functions; work better as group; sent after significant impact realized • Tier 0 – Can work remotely as situation unfolds

Avoid the scattered workforce

Communications

• Voice • Potential issues • Land lines may be out • Cell phones may be out • Solutions • Satellite phones • Private 2-way radios • IP telephony virtual phone system • Call centers • Key support teams • High risk locations

Ability to verbally communicate

Communications

• Voice • Dedicated toll-free BCP lines • Employee information line • Command conference line • Regularly scheduled, daily conferences

Ability to verbally communicate

Communications

• Data • Choose the right circuit provider • Have redundant data circuits • Different providers; different routes • Broadband wireless capabilities

Ability to access business systems

Communications

• Data • Portable satellite systems

Ability to access business systems

Communications

• Messaging • Text messaging • E-mail • Web access from anywhere • BCP web site – externally hosted

Remotely connect & send/receive updates

IT Infrastructure

• Backup power – Battery, Generator and fuel • Offsite tape rotations (e.g. Iron Mountain) • Low risk data center location • Redundant data centers • Co-location • Cold failover facility (e.g. Sungard) • Full mesh network

Just like insurance policies

IT Infrastructure PoolCorp Global Wide Area Network

Italy Branch SCP Europe MPLS E-1 1.5 megs Global Crossing Europe Portugal Branches SCP Europe MPLS E-1 1.5 megs Internet Global Crossing US European Remote Users And Customers 12 megs Global Crossing feed European Data Center MPLS E3 25 meg UK Branches SCP Europe MPLS E-1 1.5 megs Spain Branches SCP Europe MPLS E-1 1.5 megs France Branches SCP Europe MPLS E-1 1.5 megs AT&T Network Canadian Branches SCP VPN DSL 1 - 2 megs Internet 100 megs Vericenter feed MPLS DS3 25 meg MPLS OC3 155 meg US Remote Users And Customers Vericenter Dallas Data Center Covington MPLS DS3 45 meg Anahiem MPLS MLPPP 3 meg 2 T-1s Phoenix MPLS MLPPP 3 meg 2 T-1s US Branches SCP SPP Horizon Horizon South MPLS T-1 1.5 megs POOLCORP GLOBAL WAN

High speed, secure access from anywhere

IT Infrastructure

• Choose good partners • Corporate grade equipment and solutions • High reliability • Fast response time • Available technical support staff • Cost competitive • Willingness to go “above and beyond”

Only as good as the weakest link

Recovery Site Logistics

• Central command/recovery center • Wireless • Size • Proximity

From chaos to recovery in hours

Recovery Site Logistics

• Laptops for key employees • Remote connectivity (VPN) • Transportation • Housing – hotels, apartments • Childcare • Schools • Kennels (house hold pets only) • Expense reporting • Cash advances

Employees can quickly be productive

Third Party Information

• Vendors • Consultants • Financial institutions • Investors • Governmental agencies • Media • Board members

Fast access to key parties

Supplies

• First aid • Portable generators • Extension cords • Flashlights and batteries • Tarps • Tools (e.g. chain saws) • Ice coolers • Bottled water and non-perishable food • Energy drinks 

Ability to ride out DR events

Pre-event Protection Steps

• Full equipment inventory • Protect equipment and information • Unplug electronics • Move electronics off of floor • Safeguard important paperwork • Close blinds and doors • Take critical items if planned evacuation • Focus on safety if unplanned event

Attempt to minimize loss

Trigger Points

• Define for all predictable events • Example: 9-step hurricane process 1.

2.

3.

4.

5.

6.

7.

8.

9.

Storm enters gulf Projections converge with New Orleans in cone Within 4 days of landfall; still in cone Within 3 days of landfall; still in cone Within 50 hrs of landfall; still in cone; material impact imminent Within 40 hrs of landfall; still in cone; material impact imminent Within 30 hrs of landfall; still in cone; material impact imminent Next 36 hrs during/after storm Authorities give “all clear” to return home

Know what to do and when to do it

Municipal, State, Federal Interaction

• Participate in municipal DR planning/testing • Get to know local and state officials • Know evacuation routes • Placards to re-enter impact areas

Take advantage of available help

How to Implement

• Start with basics • Focus on critical systems, functions, people • Use available “free” help and templates • Hire consultants if needed • Train and communicate

Audit

• Annual testing • Validation of tape backups • Failover to backup facility • User validation • Signed acceptance forms

Create Don’t be overwhelmed

Effort Required

• 300-400 hours can get you started • 100-200 hours annually to test/audit • 100-200 hours annually to enhance/update • 5 person “core team” • Senior Management Sponsor • Project manager • Tech writer • IT manager • Logistics/facilities coordinator

It can be a reasonable effort

Value Obtained – Basic BCP Plan

• Creating the initial plan: • Annual updates and testing: • Annual infrastructure costs: • Peace of mind during a disaster: $15-20K $10-20K $200K Priceless

The best insurance policy you’ll ever buy!

Tim Babco (985)801-5230 [email protected]