Transcript Document

Behind-the-Scenes at
Salesforce.com
R&D: Powering 150 180+ Million Transactions a Day
Claus Moldt, Salesforce.com
Safe Harbor Statement
“Safe harbor” statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forwardlooking statements including but not limited to statements concerning the potential market for our existing service offerings
and future offerings. All of our forward looking statements involve risks, uncertainties and assumptions. If any such risks or
uncertainties materialize or if any of the assumptions proves incorrect, our results could differ materially from the results
expressed or implied by the forward-looking statements we make.
The risks and uncertainties referred to above include - but are not limited to - risks associated with possible fluctuations in
our operating results and cash flows, rate of growth and anticipated revenue run rate, errors, interruptions or delays in our
service or our Web hosting, our new business model, our history of operating losses, the possibility that we will not remain
profitable, breach of our security measures, the emerging market in which we operate, our relatively limited operating
history, our ability to hire, retain and motivate our employees and manage our growth, competition, our ability to continue to
release and gain customer acceptance of new and improved versions of our service, customer and partner acceptance of
the AppExchange, successful customer deployment and utilization of our services, unanticipated changes in our effective
tax rate, fluctuations in the number of shares outstanding, the price of such shares, foreign currency exchange rates and
interest rates.
Further information on these and other factors that could affect our financial results is included in the reports on Forms 10K, 10-Q and 8-K and in other filings we make with the Securities and Exchange Commission from time to time. These
documents are available on the SEC Filings section of the Investor Information section of our website at
www.salesforce.com/investor. Salesforce.com, inc. assumes no obligation and does not intend to update these forwardlooking statements, except as required by law.
Claus Moldt
VP, Technical Operations
Data Centers
Best of Breed Data Centers
Fully Mirrored Cloud Computing Infrastructure
Continued Investments. Unparalleled Confidence.
 Unmatched Reliability
 Two (soon to be 3) mirrored
production data centers plus a
production-class lab facility
 Near real time replication
between facilities
 Validated disaster recovery
 MPLS based backbone
 Maximum Uptime & Performance
 Trusted Security
 Carrier neutral network strategy
 World-class security specs
 No single points of failure
 SAS 70 Type II and SysTrust
Certified
 Carrier level scalability
 Extensive use of high availability
server and network technologies
 ISO 27001 Certified
 Secure point-to-point data
replication
 Secure custody of customer
data and backups
Production-Class R&D Lab
& Tape Archive (CA)
Asia Pacific Production
Datacenter (Singapore
Winter ‘08)
Main Production
Data Center (CA)
Back-Up Production
Data Center (VA)
Security: Facilities
Maximum Facilities Security
 24 x 365 on-site security
 All doors, including cages, are secured with biometric hand
geometry readers.
 Five levels of biometric scanning including man-traps required
to reach Salesforce cages
 Fully anonymous exteriors
 Digital camera (CCTV) coverage of entire facility
 Entire perimeter bounded by concrete bollards/planters
 A silent alarm and automatic notification of appropriate law
enforcement officials protect all exterior entrances.
 CCTV integrated with access control and alarm system.
 Motion-detection for lighting and CCTV coverage.
World-Class Infrastructure
Delivering leading On-Demand availability
 Two mirrored data centers plus a production-scale lab facility
–
18,000 total sq. feet of cage space
–
Mirroring is about more than just having a copy of your data
–
Salesforce maintains a full-scale replica of the production facility as well
as your data
 Power: Diesel Generators for backup power supply

Next generation UPS systems (N+1)


Five- Hitec Rotary Continuous Power Supplies rated for 4,980kW (n +1)
Rotating fly-wheel generator provides UPS and Diesel generator start-up

Two- Detroit Diesel engine 2mW Generators for a total of 4,980kW (n +1)

Eliminates potentially risky UPS battery maintenance

25,000 gallon diesel fuel tanks supported by two fuel vendors
 Cooling
–
Precision, N+1 HVAC
–
Guaranteed by backup water supply
–
On-site dedicated wells
Network
Industry leading performance, scalability and redundancy
 Carrier-class and carrier-neutral model: multiple transit
vendors

AboveNet

MCI

Level 3

NTT

Equinix Exchange

Sprint
 Multi-gigabit IP transit for external customer service
 Lightning-fast performance worldwide

Data centers located at core Internet hubs

Access to thousands of global Internet peering points delivering
global high performance access

Private peering with key carriers and partners (15+)
 MPLS/VPLS based backbone

Enables near real-time replication for availability and disaster
recovery
Scalability
Highly Scalable POD Architecture
Cloud Computing Serves Companies of All Sizes
ENTERPRISE
MARKET
Enterprise Std
MID-MARKET
~9,000
SMALL BUSINESS
~4,000
~65,000
~30,000
~30,000
~6,300
~5,800
~5,500
~3,500
~3,200
Number of Subscribers
~3,000
We built the platform for the cloud
We do
Infrastructure
Services
We do
Application
Services
We do
Operations
Services
YOU
get to focus on
innovation
Network
Security
Authentication
Build your data model
Storage
Sharing
Availability
Operating System
Integration
Monitoring
Database
Customization
Patch Mgmt
App Server
Web Services
Upgrades
Web Server
API
Backup
Data Center
Multi-Language
NOC
Build your business logic
Build your user interface
Force.com allowed us to create and deliver a total of 14
“
applications – all without the expense and hassles of traditional
application development.
”
The Cloud Computing Model: Multi-tenant, Subscriptions
Subscription
Multi-tenant
Faster Vendor Innovation
Economies of Scale
Scalability
Automatic Upgrades
The Fastest, Easiest and Lowest
Risk Path to IT Success
Client/Server & App
Server Platforms
Source: 3rd party analyst surveys
Platform as a Service
Source Salesforce.com Customer Relationship Survey conducted in
Feb. 2008, by an independent third-party CustomerSat Inc.
Cloud Computing Enables Reactive Innovation
26 Major Releases in 9 Years
No Customers left behind
Every customer on the latest
version of salesforce.com
All Customizations Upgraded
Automatically
Proven Scalability and Performance
Delivering 180+ Million Transactions Daily
12.0
11.0
1,250
10.0
9.0
Quarterly
Transactions
(billions)
1,000
8.0
7.0
750
6.0
5.0
500
4.0
3.0
250
2.0
1.0
0
0.0
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2
Fiscal Year
2005
2006
2007
2008
Page
Response
Time(ms)
Multi-Tenant Integration = Proven Success
Over 2.2 Billion API Transactions per Month
API Transactions
3,000,000,000
2,750,000,000
Page Views
2,500,000,000
2,250,000,000
I think API font should be bold
2,000,000,000
and line stronger. Make page
views gray or something to
highlight that API transactions
1,750,000,000
is the key thing to focus on.
1,100,000+
Subscribers
1,500,000,000
1,250,000,000
1,000,000,000
750,000,000
Q2FY06
Q3FY06
Q4FY06
Q1FY07
Q2FY07
Q3FY07
Q4FY07
Q1FY08
Enterprise Scalability & Performance
1
Your Company
CEO
VP Sales
Director Sales
West
2
VP Customer
Service
VP Marketing
Director Sales
East
Director Customer
Service
COO
Director Support
VP Professional
Services
East Services
West Services
Your Division
CEO
VP Sales
Director Sales
West
VP Marketing
Director Sales East
VP Customer
Service
Director Customer
Service
Director Support
COO
VP Professional
Services
East Services
West Services
3
Your Customizations
4
Your Sharing Model
Query
Optimization
Engine
Your
Data
Massive Scale
Immediate Response
Billions of
Transactions
Sub-second
response time
Scalable Software Architecture
Utilizing Industry Standard Platforms for High Availability
 Database Server: Oracle RAC EE, Dell, Sun
 Clustering: SunCluster
 Web Site and Application Server: Dell, Resin
 Search Server: Jakarta Lucene
 Storage Management: Hitachi Data Systems,
Sun
 Backup Software: Veritas/RMAN
 Operating Systems
– Sun Solaris
– Redhat Linux
Pod Architecture further enhances availability,
horizontal scale, and platform for future growth
APAC
Pod
NA0
Pod
NA1
Pod
Network
Services
NA2
Pod
Storage
Services
EMEA2
Pod
Backup
Services
NA6
Pod
NA3
Pod
NA4
Pod
Monitoring
Services
Sandbox
Pod
NA7
Pod
NA5
Pod
Threshold User Capacity = Add a POD
EMEA
Pod
“N”
Pod
Salesforce.com confidential
What’s Shared Across Pods

Storage – HDS 9990

SAN – Cisco MDS
– syslog

Core Network – Force10
– bastion

Edge Network – Juniper

Search Indexer – Sun SPARC

WWW Services – Dell/Linux

Edge Firewalls – Juniper
Netscreens

Load Balancers – F5
– TACACS

Proxy Services – Dell/Linux
– SecurID

IDS

BlueCoat

Performance Monitoring

Email
 Ops Stack
Salesforce.com confidential
– jump/kick start
– release
– backup
– DNS
Redundancy
Network Redundancy/Multiple
Carriers
Load Balancing/Fail-Over
Clustering/RAC EE
MirrorForce
SFDC Built for High Availability
– Multiple Network Carriers
– Redundant Routers at Entry Points
– Fail-over Configured Firewalls
– Redundant & Load Balanced Load Balancers
– Redundant Hubs/Switches at VLANs
– Web, Application, API, Search, Cache, Index, Batch Servers
• Load Balanced, Fail-over or Clustered
– Data Base Servers
• Oracle RAC EE running on 4 way Clustered Nodes
• Sized to sustain Peak Load if Node failure
– Storage
• Multiple paths for reliability
– 4 inter-connects per DBMS Server
» Alternate paths to separate Storage Directors
– 2 Storage Directors per Array
Multiple Network Carriers and Redundancy at the Edge
Equinix
Exchange
AboveNet
NTT
Level3
Sprint
MCI
Edge Routers
Edge Routers
Redundant
Firewalls
Redundant
Load Balancers
Redundant
Core Switches
Si
Si
Server VLAN Switches/Firewalls
Sample POD Architecture – Built for Redundancy
Backup and Disaster Recovery Strategy
 Near real time
replication between
data centers
 Disaster Recovery
Strategy: Failover to
full-scale east coast
replica data center
backup facility.
Near real time
replication between
data centers
West Coast
Production
Data Center
East Coast
Backup DR
Data Center
OC48/MPLS/VPLS
Backbone
Lab and Tape
Archive
* Local 48 Hour lag
standby databases
San Francisco
Monitoring
Performance Management
Pro-active Monitoring
Performance and Monitoring
End-to-End Monitoring Guarantees Uptime and Security
 Monitoring Strategy: Multi-Tier Monitoring Strategy
 Nagios monitoring software
 Gomez performance software service
 EMC Smarts
 Coradiant End-User experience
 Custom instrumentation within the Application
 Performance Metrics
 Average page load times between 250 and 400
milliseconds
 180M+ Transactions Daily
Cricket
 Pages served in fiscal Q1 2008: 5.4 Billion
 47,600+ Customers
 1,100,000+ Subscribers
Custom Agents
 95% Customer Satisfaction*
 Open Communication
 http://trust.salesforce.com
*January 2005 independent survey
Example of SFDC Monitoring Pro-Agents

Trust Site - Incident Communications Example
Users on NAX instance may experience latency with Dashboard Refresh, Reporting, and Customer Self Service Portal. The Salesforce.com
Technology team is actively working to resolve these issues. Please check back for latest update
.
Trust Site - Security Alerts and Examples
Capacity Planning
Capacity Planning and Analysis Model
System/App
Outputs
Log Parsing and Analysis
Engine
 Basic load management
data used to forecast
annual growth.
 Detailed customer
transaction level
monitoring for focused
and proactive capacity
management
 Granular performance
breakdowns by transaction
Breakdown of Customer
type
Transactions
Operations
Data Store
Capacity forecasting
 Impact analysis of custom
transactional logics
 Operationally efficient and
scalable
Multiple detailed metrics
1. Annual demand growth
for all enterprise customers
are projected and reviewed
every week
2. Three year forecast are
predicted for Datacenter
capacity
3. Results analyzed and
systems scaled
appropriately to meet
demand growth.
Security
Overview
Security
 Dedicated Security Organization
 Strategy/Charter



Mitigate risks while complying with legal, statutory, contractual, and internally
developed requirements
Develop and enforce policies and procedures
– Design and secure information systems using
security domains, defense in-depth and least privilege principles
– Develop and integrate security architecture into business processes (CobiT,
ISO27001)
– Conduct employee security awareness training classes
– Perform regular vulnerability assessments and audits
Addresses all layers
–
–
–
–
–
Physical Security
Logical Network Security
Host Security
Transmission Level Security
Database Security
Internal Vulnerability Assessments

Salesforce.com implements a multi-prong approach to ensure the software we
release is secure. Specifically, we perform the following tasks to assure security in
the development lifecycle.




Architecture Reviews Salesforce.com architects (including security team)
meet regularly to discuss features that could be considered high risk.
Development Salesforce.com developers follow coding best practices such as
those specified in OWASP. All code prior to check in is reviewed. Code quality
and security tools (Findbugs, Checkmarx.) are run frequently to detect
possible program anomalies. All developers receive application security
training to help them write secure code.
Quality Assurance Salesforce.com QA testers analyze their features through
both positive and negative testing. Salesforce.com also employs several black
box analysis tools (Appscan, Peros, etc.) to help in identification of security
vulnerabilities.
Information Security Salesforce.com InfoSec tests medium and high risk
features. (Proprietary fuzzers, Burp Suite) Periodically brings in third parties
to perform code reviews, blackbox analysis and design reviews (iSEC Partners,
etc.)
External Vulnerability Assessments

MSSPs include SPI Dynamics, Solutionary, Symantec

Network Assessments and Application Assessments

Assessments cover the following:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Cross-Site Scripting
Input validation
Buffer Overflow
SQL Injection
Directory Traversal
Parameter Overflow
Path Manipulation
Command Execution
Path Truncation
Character Encoding
Character Stripping
Site Search
Application Mapping
Automatic Form-Filling
Configuration Management
Proxy Support
Parameter Injection
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Directory Enumeration
Authentication and Session Management
Web Server Assessment
HTTP Compliance
SSL Support and Strength
Certificate Analysis
Content Investigation
Spam Gateway Detection
Developer Comments
Absolute Path Detection
Error Handling
Permissions Assessment
Brute Force Authentication attacks
Known Attacks
Session Hijacking
Horizontal Attacks
Insecure Storage
Executive Summaries available upon request
Managing Change
Release Management/Change
Management
Maintenance Windows
SFDC Release Testing/Managing Quality & Change
 SFDC Testing is focused to ensure transparency of
changes
– Intense Functional and System Testing prior to release
– Forward and backward compatibility of all standard API’s
– Review Teams
– Metrics and Reporting
• Quality Targets
• System metric/trends
 All production changes logged in cases
– Includes rollback, validation and expected impact
Salesforce.com Releases
Release
Release objective
Frequency
TYPICAL Time of
the week
Example
Major
Significant new
functionality and
enhancement
Approx 3 - 4 per year
Friday night,
Saturday
Planned 146 release
Patch/Dot
Bug fixes or minor
functionality
enhancement
Weekly for first 3-4
weeks after major
release
Every other week
there after
Wednesday evening
(No Downtime)
End user experience
enhancement with
146.8 release
Fix production
vulnerabilities
Unscheduled on as
needed basis
(No Downtime)
Break fix errors
E Release
Salesforce.com confidential
Maintenance Windows are Designed to Minimize
Business Disruption to Customers
Reserved
Declared
Actual

Established based on analysis of our customer
usage patterns and traffic

4 hour windows reserved for routine maintenance
~4 Hours
~30 Minutes
~15 Minutes

–
1st & 3rd Saturdays
–
7pm to 11pm Pacific Time all NA & EU Instances
except NA2
–
12am to 0400am Sunday Pacific Time NA2 only
–
10am to 2pm Saturday Pacific Time AP0 only
–
Plans for EMEA instance can be adjusted to fit their time
zones (for maintenance of non-shared infrastructure)
–
Maintenance of Shared Infrastructure 1st & 3rd
Saturdays 7pm to 11pm Pacific Time
–
Windows are planned conservatively
–
Not all reserved windows are utilized
–
Actual maintenance downtime is a fraction of declared
window
Future roadmap to minimize and eventually eliminate
downtime
Note: Product release updates (3 per year) typically occur on a separate schedule on Friday nights and have longer windows
Maintenance Windows are Declared 1 Week in
Advance
Sample Notification
Thank You
Q&A