Transcript DTCC’s Response to Super Storm Sandy Cecilia Humphrey Americas’ Representative, DTCC Special thanks to: David.

DTCC’s Response to
Super Storm Sandy
Cecilia Humphrey
Americas’ Representative, DTCC
Special thanks to: David. LaFalce, Business Continuity Management; and
Michael Obiedzinkski, Enterprise Communications
ACSDA General Assembly, April 2013
Antigua, Guatemala
DTCC Green
DTCC’s Sandy Recovery: ‘Why It Worked’
DTCC Green
2
Business Continuity Management (BCM)
Preparations – Pre-Sandy
Business Continuity Management
preparation includes:
• Consistent execution of BCM tests -announced and unannounced
• Cross-training and transfer of work
functions (e.g. stand-down
sustainability exercises, remote
access) across sites
• Well-documented procedures and job
aides for critical products/processes
• Rotation of operational responsibility
for product and technology operations
• Activation of emergency notification
systems to ensure information sharing
DTCC Green
Prior to Sandy, DTCC’s facilities were
operating as follows:
• Staff located in DTCC offices in New
York, Tampa, Dallas, London, Shanghai,
Manila, Chennai, and Pune
• DTCC Dallas operating as backup for
technology
• DTCC Tampa backing up business
operations
• DTCC Brooklyn operating as primary data
center, with synchronous replication to
DTCC Manhattan and asynchronous
replication to DTCC Dallas
• Only production data was being
replicated between sites
• Initial Work-from-Anywhere (WFA)
capabilities enabled through Virtual
Desktop Infrastructure (VDI) established
in preparation for move to DTCC Jersey
City
3
BCM Response to Sandy
Friday- October 26: Corporate Command Team activated pre-Sandy.
Saturday- October 27: State of Emergency declared in New York.
Sunday- October 28: Production operations transferred to Tampa. Technology operations
transferred to Dallas.
Monday- October 29:
–
–
DTCC processing fully operational.
Disaster declared at 7:45pm by NYS. All core production systems recovered within 2 hours and transferred to
DTCC Brooklyn facility.
Tuesday- October 30: Markets closed. DTCC processing fully operational.
Wednesday- October 31: Markets re-open. 300 DTCC employees redeployed to Brooklyn.
Thursday- November 1: Securities processing for physical securities established (Brooklyn).
Over 400 employees now operating out of DTCC Brooklyn.
Friday- November 2: Physical document recovery plan/relocation plan takes shape.
November 5-9: 650 employees in DTCC Brooklyn. Volumes return to normal levels.
November 12-16: Decision to accelerate move to DTCC Jersey City. Ongoing recovery planning
for DTCC Manhattan.
November 19-23: Vault recovery begins. Certificates begin to be packed and prepared for
restoration.
November 26 to Present: Vault recovery completed. 1,500+ employees relocated to DTCC
Jersey City. DTCC Manhattan normalized.
DTCC Green
4
Post-Sandy Recap-- Areas of Success
•
•
•
•
•
•
•
•
Successful execution of business continuity and disaster recovery plans
– Transfer and dispersal of core services
Global support from teams in Tampa, Dallas, London , Shanghai and Chennai
Restoration of employee connectivity and functionality
– Work-from-Anywhere program, and accelerated DTCC’s Jersey City move.
Re-creation of compromised capabilities, e.g., IT developers’ desktop tools and
test environment for GTR Participant Testing
Leveraging vendor relationships minimized costs and loss of productivity
• Temporary rental of equipment kept operations running, backed up
• Creation of an innovative service-credit arrangement, i.e., switched some
processing to optimize equipment efficiency
Continuous processing of transactions
Proactive and frequent communications with regulators, clients and employees
After years of preparation, well practiced plans with defined roles and
responsibilities were activated, with little to no need for improvisation
DTCC Green
5
Next Steps- Governance
•
•
•
•
•
Address concentration risk and the
potential problems that a similar or
larger storm might cause:
• Regional exposure in the Northeast
due to having 3 sites in the region
• Formally measure and address
location risk and all viable options
Reassess footprint drivers; regulatory,
business, technology, financial, and
operational
Evaluation of business needs to ensure
recovery prioritization is correct
Expand business continuity to include
plans to remotely maintain all
processing and operations for an
extended period of time
Achieve dematerialization for physical
securities – this will reduce risk for all
parties involved
DTCC Green
•
Keep in mind that different
products may have different
timing for risk tolerances
0 hours
4 hours
Next day
> 5 days
2 hours
3 days
6
Next Steps- Planning
•
•
•
Develop a more comprehensive inventory
of all assets across people, places, and
• Integrate plans to include
things
recovery of people, places
Continue to ensure we have all the required
and things
skills, resources and capabilities to
seamlessly handle operations and
functions across all sites
Enhance general logistics and procedures;
including up-to-date mailing and phone
number lists and ensuring employees have
all equipment necessary for remote access
as well as have all sign-in information
available
DTCC Green
7
Next Steps- More Testing
•
•
•
Enhance crisis management
and testing
Leverage the emergency
notification tool for employee
status checks during tests
Transition from planning and
testing for a “single type of
impact” catastrophe to a more
variable set of impacts
DTCC Green
•Create the ability to test end-to- end
8