Transcript Opportunity: IIM and Standard Placement API

Emerging Practices and Standards for
Information Lifecycle Management
Bill Pierce
SNIA, TeraCloud Corp.
September 14, 2005
Outline
• What is ILM?
• SNIA’s ILM Modeling and Standards
• Implementing ILM at a Small Software
Company
• Reactions?
Property of SNIA
2
SNIA Definition of ILM
The policies, processes, practices, services and
tools used to align the business value of
information with the most appropriate and costeffective infrastructure from the time information
is created through its final disposition.
Information is aligned with business
requirements through management policies and
service levels associated with applications,
metadata and data.
Property of SNIA
3
The Problem
•
•
•
•
•
Information stewardship falls mostly on IT
Data placed for convenience
Too many configurations to support
Nonstandard, unreliable results
Fails to take advantage of economies of
scale
Property of SNIA
4
Business Goals of ILM
• Reduce the cost of managing storage
services
• Obtain an objective best fit between data
requirements and storage services
• Ensure compliance with corporate
governance and government regulation
Property of SNIA
5
SNIA’s role in ILM evolution
• This is work in progress!
• Technical Working Groups
– Architecture
– Standards
– Product interoperability
• Marketing Forums
–
–
–
–
Requirements analysis
Use cases
Best practices
Education and certification
Property of SNIA
6
Data Management Forum: Use Cases
• Primary Archetypes
– Business Process
Analyst (business
requirements)
– IT Architect (makes it
happen)
– Records Information
Manager (regulatory
requirements)
Property of SNIA
7
ILM in Systems Administration
• Just as systems administration is the process of
applying design and planning to meet computing
application requirements at least cost, ILM is a
similar process focused on information
requirements.
• ILM provides a language for business, IT and
records management to share the responsibility
of information stewardship.
Property of SNIA
8
ILM TWG Modeling: (work in progress!)
•
•
•
•
•
Data Classification
Data Lifecycle Management
Composite Storage Services
Storage Service Level Management
Data Placement Services
Property of SNIA
9
Data Service Resource Mgmt
Data requirements
• Data classification bridge to
Information-based mgmt
• Service Level Management
• Lifecycle Management
Data Service Resource Mgr
Mgmt
Data Services
Data Center Management
Data
Security
Service
Mgmt
Data
Protection
Service
Mgmt
Data
Placement
Service
Mgmt
Storage & Storage Services
Property of SNIA
Mgmt
Composite Storage Set
Mgmt
Storage solutions
• Configuration templates
• Resource Management
• Management of individual
services
Business Applications
10
Information
Lifecycle
Information Classification
Collaboration
Key Performance
Indicators (KPIs)
Business
Data Center
Data Lifecycle
SNIA will define standards for:
- Data SLOs
- Offered Data Svc Levels
- Data Lifecycle Policy Management
- Composite storage configs
Standard Configurations
- Mgmt of underlying
- Storage
- Data Protection
storage services
- Data Security
Property of SNIA
Data Service
Level Objectives
Data Classification
& Lifecycle
automation policies
11
Data Classification
Data
Data
SLOs
Data Classification
Policies that group data so as to determine what
Service Level Objectives must be applied.
Examples:
• Files in /usr/foo/bar
• Files in /user/fred older than 3 months
• Files named *.mp3
• Files bigger than 1GB
• Files originated by a corporate officer
• Files in the engineering group
• Files with keywords in content
• Files about oil exploration
•…
Property of SNIA
12
Data Classification Model
PolicySetAppliesToElement
ManagedElement
-Metadata
-//LogicalFile, Volume, or object
*
*
ClassificationSchemeAppliesToElement
MemberOfCollection
PolicySetAppliesToElement
Classification : PolicySet
1
Data : ClassificationScheme
-ClassificationSchemeName : string
-Comments : string [ ]
1
MemberOfCollection
Classified : ClassificationNode
-ClassificationName
1
*
ElementCapabilities
ServiceAffectsElement
ClassificationService
Data Classification
Service Instance
ClassificationCapabilities
Property of SNIA
13
Standard Metadata org.snia.fsmtwg.*
Name
Name
FileSize
CreationDate
Type
Uint64
Datetime
Description/Values
Full path name
Size in bytes
File creation time – nfs does not support this
LastModified
Datetime
File modification time
LastAccessed
Datetime
File last access time
EncryptionMethod
String
Describes the method used to encrypt the file
InUseCount
Uint64
Used like a semaphore to determine when the file is in use
Remainder
Of
CIM_LogicalFile attributes
Property of SNIA
14
Standard attributes org.snia.ilmtwg.*
Name
Type
Description/Values
AccessingApp
String []
Data was opened by
CreatedByApp
string
The name of the application that created the data
BytesUsed
Uint64
The number of bytes actually used to hold the data
FileURL
String
The URL of the file
ObjectID
String
A unique identifier for this data object (XUID**)
ObjectType
Enum
File, Directory, (stat data ++) XSET**
Version
string
The version of the document this data object represents
UserDefined
String
A free-form text comment
RetentionStartDate*
Datetime
The date on which the retention period begins
RetentionInterval*
Int64
RetentionEventDate*
Datetime
The amount of time after the RetentionStartDate to retain
the data.
The date on which an event occurred
RetentionEventInterval*
int64
The number of seconds after RetentionEventDate for which
to retain the data
ExpirationHold*
Boolean [ ]
All holds must be cleared in order to expire
RetentionEnabled*
Boolean [ ]
Declares whether retention attributes are enabled and/or
infinite
Property of SNIA
15
Data Lifecycle Management
Data Lifecycle Management:
Policies that define Service Level Objectives for data over its lifetime.
Data
Data
SLO
Policy
Property of SNIA
Data
SLO
Policy
Data
SLO
Policy
16
Data Service Level Objectives (examples)
• Budget - MaxBudget
• Accessibility
– ReadWriteRatio, AvgDataThroughput, SpaceUsage,
SizeGrowth, Location
• Availability
– PlannedDowntime, MaxUnplannedDowntime,
Aggregate, MaxUnplannedDowntimePerInstance
• Data Restore – RPO, RTO
• Security – Expiration, DestructionType
Property of SNIA
17
Composite Storage Service
ElementSettingData
*
CSSStorageSettings
CompositeStorageSet
CSSSettingData
-CssName : string
-StorageResourceType : enum(diskblock..files)
-Owner : String
1
-Tariff : Integer
-ResourceClassification : String
ConcreteComponent
0..1
0..1
-Workload : string [ ]
-AvgThroughput : integer [ ]
-AvgDataTransferRate : integer [ ]
-MeasuredCapacity : integer [ ]
-Protocol : string [ ]
-InitialAccessTime : enum(immediate..hours) [ ]
CSSSecuritySettings
ServiceAffectsElement
CSS
ServiceAffectsElement
-SecurityAccountability : enum(none..full)
-SecurityIntegrity : enum(none..full)
-SecurityAuthenticity : boolean
-SecurityTrustworthiness : boolean
-SecurityConfidentiality : enum(none..full)
-SecurityImmutability : boolean
-SecurityDestruction : boolean
-SecurityPhysical : enum(none..full)
CompositeStorageConfigurationService
MemberOfCollection
MemberOfCollection
CSSProtectionSettings
+CreateCSSSettingDataTemplate()
+DestroyCSSSettingData()
+CreateCompositeStorageSet()
+ReclassifyCompositeStorageSet()
+ClassifyResource()
-Workload : string
-RecoveryCopyFrequency : enum(cont..weekly)
-CopyRestoreTime : integer(seconds)
-MeasuredCapacity : integer(bytes)
-MaxCopyRetentionCapability : integer(seconds)
-DataCopyLocale : string
Service
Included in Composite Storage Service profile
*
StorageVolume
Property of SNIA
Configuration template
*
FileShare
18
Composite Storage Set Attributes (examples)
• Storage
– Workload, AvgThroughput, InitialAccessTime
• Protection
– RPO,DataRestoreTime,MaxDataRetentionCa
pability,DataCopyLocale
• Security
– Accountability, Integrity, Physical, Destruction
Property of SNIA
19
CSS, ODSL, and SLOs: Service Level Mgmt
Data Service Level Objectives
Derived from requirements defined in
Information Classification processes.
Includes requirements specific to the
data and an application’s use of data.
ODSL: Offered Data Service Level
Part of the “service catalogue.” Each service
classification describes the service level
that can be provided to data.
Data
Data
SLOs
Map to best fit
Service
Classification
Properties
Map per data center policies
CSS: Composite Storage Set
An abstract storage configuration template
described by its overall technical capabilities
such as storage performance, availability,
data protection and data security.
Property of SNIA
DR
Capabilities
20
CSS, ODSL, and SLOs: why 3 abstractions?
Data Service Level Objectives
Requirements for data may or may not
change independent of service catalogue.
Data
Data
SLOs
Map to best fit
ODSL: Offered Data Service Level
Service catalog may remain stable while
supporting configurations evolve.
Service
Classification
Properties
Map per data center policies
CSS: Composite Storage Set
Supported configurations evolve with technology
towards improving costs while meeting existing
service requirements.
DR
Capabilities
Property of SNIA
21
SMIS 1.3 –
Expanded
Data Services:
Data
Classification
DataObject : ManagedElement
-Metadata
PolicySetAppliesToElement -/From LogicalFile, Volume, FileShare, Dir, XSystem or XSET
*
2 - MemberOfCollection
3 - LifecycleAppliesToElement
1 - ClassificationSchemeAppliesToElement
PolicySetAppliesToElement
Classification : PolicySet
Data : ClassificationScheme
-ClassificationSchemeName : string
-Comments : string [ ]
1
Data : ClassificationNode
MemberOfCollection -ClassificationName
*
1
LifecycleSelection : PolicySet
Data
4 - PhaseAppliesToElement
Lifecycle
Management
PolicySetAppliesToElement
LifecycleClassification
PhaseSelection : PolicySet
PolicySetAppliesToElement
1
*
Blue : Lifecycle
MemberOfCollection
Phase1 : Phase
ElementSettingData
PhaseMapsToODSL
Data
Service Level
Management
Gold : OfferedDataServiceLevel
-ServiceLevelName : String
SettingsDefineCapabilities
ServiceLevelMapsToResource
DataServiceCapabilities
CSS
Property of SNIA
DataSLOGs
-SLOAttributes
-//To be expanded
-DataSLOAttributes
-// to be expanded
22
Implementing ILM at a Small Software Company
1)
2)
3)
4)
5)
6)
7)
8)
9)
Classify by Business Process Application
Define Lifecycles and SLOs
Define Lifecycles for backup copies
Classify storage or define ODSLs (<~10)
Match Phase SLOGs to ODSLs
Configure CSSs
Match ODSLs to CSSs
Implement Lifecycles
Operational Adjustments
Property of SNIA
23
Classify by Business Process Application
•
•
•
•
•
•
•
•
•
•
Project documents
Software archive
Windows Updates
Financial spreadsheets
Customer contracts
Home directories
Web site
Released build archive
Accounting database
File transfer
Property of SNIA
•
•
•
•
•
•
•
•
•
•
OS Software
OS Images
Wiki Site
Corporate Documents
Recent Builds
Source code files
Source Control Database
Exchange Email
Temp Files
Backups
24
Try to identify common superclasses
• Working Documents
– Project documents, Recent builds, Web site
• Archives
– Software archive, OS images, Released build archive
• Database Managed Data
– Exchange Email, Source Control Database
• Sensitive Data
– Customer contracts, Accounting database
• Temporary
– Temp files
Property of SNIA
25
Define Lifecycles and SLOs
Working Docs Lifecycle
if not accessed
in 6 months
if not accessed
in 2 years
if not accessed
in 8 years
Working Docs: Tier1
Working Docs: Tier2
Offline
AvgIORate:Medium
AccessPattern: Sequential
Size: Medium
SizeGrowth: Medium
MaxDowntimeAggregate: 2 day
MaxDowntimeInstance: 1/4 day
RPO:1 day
RTO: 1 day
Security: Normal
AvgIORate: Low
AccessPattern: Sequential
Size: Large
SizeGrowth: Medium
MaxDowntimeAggregate: 5 day
MaxDowntimeInstance: 1 day
RPO: 1 week
RTO: 2 days
Security: Normal
Initial Access Time: 1 day
Location Affinity: Onsite;
Offsite Copy
Security: Normal
If accesed in <
6 months
Property of SNIA
26
Define Lifecycles and SLOs
Database Managed Lifecycle
Database: Tier1
AvgIORate:High
AccessPattern: Random
Size: Large
SizeGrowth: Large
MaxDowntimeAggregate: 2 day
MaxDowntimeInstance: 1/4 day
RPO:1/4 day
RTO: 1/2 day
Security: Elevated
lifecycle
managed by
DBA
Database: Backup
backup
Property of SNIA
27
Define Lifecycles for Backup Copies
• Will not discuss (much)
• These copies have their own lifecycles
and requirements in each phase
• Includes need for technology refresh
– Physical (worn out media)
– Logical (antiquated data formats)
Property of SNIA
28
Define ODSLs
Software Company ODSLs
Active
Archive
Sensitive
AvgIORate:High
AvgThroughput: High
AccessPattern: Random
Size: Large
SizeGrowth: Moderate
MaxDowntimeAggregate: 2 day
MaxDowntimeInstance: 1/4 day
RPO:1/4 day
RTO: 1/2 day
Security: Elevated
AvgIORate:Low
AccessPattern: Sequential
Size: Large
SizeGrowth: Large
MaxDowntimeAggregate: 5 day
MaxDowntimeInstance: 1 day
RPO:2 day
RTO: 2 day
Security: Normal
AvgIORate:Medium
AccessPattern: Random
Size: Small
SizeGrowth: Small
MaxDowntimeAggregate: 2 day
MaxDowntimeInstance: 1/4 day
RPO:1/2day
RTO: 1/2 day
Security: Secure Delete
DestructionType: Secure
Property of SNIA
29
Best Practices for ODSLs
• You are trying to simplify and reduce the
number of ODSLs you provide.
• There is a strong tendency to overoptimize each storage solution. Resist it.
• In the name of simplicity, you will have to
over-provide for some classes of data
Property of SNIA
30
Match Phase SLOGs to ODSLs
Class
Working Docs
Phase
Tier 1
ODSL
Active
Working Docs
Archives
Database
Tier 2
Tier 1
Tier 1
Archive
Archive
Active
Sensitive Data
Sensitive Data
Temporary
Tier 1
Tier 2
Tier1
Sensitive
Sensitive
Archive
Property of SNIA
31
Configure Composite Storage Sets
• This is where a lot of practical
considerations come in.
– Space and throughput limitations
– Budget
– Technology (old and new)
– Physical and logical access
– Vendors
• SRM tools can help
Property of SNIA
32
Match ODSLs to CSSs
• Probably a manual process
• With standardization, more automation
Property of SNIA
33
Implement Lifecycles
• Work in progress…
• Includes initial data placement
• Technologies available
– Scripting and Symbolic Links
– HSM and SAN Filesystems
– DMAPI
Property of SNIA
34
Operational Adjustments
• Further economies may be achieved by
splitting or combining classes and ODSLs
• New applications come online
• New technologies become available
• With ILM you have the infrastructure to
deal with these changes in a coherent way
Property of SNIA
35
Summary
• ILM brings information storage in line with
the requirements of the data
• Almost any IT organization can benefit
from some aspects of ILM practice
• SNIA is laying the foundation of ILM
through the development of Best Practices
and Standards
Property of SNIA
36
For more information…
• www.snia-dmf.org
• www.snia.org
Property of SNIA
37
Questions? Reactions?
• Was this:
– Useful?
– Obvious?
– Impractical?
– Something you are already doing?
– In need of standardization?
Property of SNIA
38
Abstract
Information Lifecycle Management (ILM) is an effort to formally
align the business requirements of data with the storage services
that support it. It is not a product, but a process that could benefit
most organizations. Failure to do this can result in lost data, stolen
data, unavailable data, overly expensive data and too much
useless data. The Storage Networking Industry Association (SNIA)
is working to define Best Practices and standards to support ILM.
This talk will describe ongoing work in the SNIA and its application
to the data and storage resources at a small software company.
Property of SNIA
39
Bio
Dr. Bill Pierce started his career as an experimental physicist
at Cornell University, the University of Washington and
Northwest Research Associates, Inc. In 1995, he turned to
systems administration and software engineering, supporting
scientific projects during the early growth of the Internet.
From 1998-2002 Bill worked on one of the first SAN
Management applications at Vixel Corp. Since 2002 he has
been actively involved in the Storage Networking Industry
Association and the development of Information Lifecycle
Management at TeraCloud Corp. Bill is the author of the
Open Source storage utilities fcping and io_profile and
numerous scientific and technical articles. He is a member of
the Seattle Area System Administrators Guild and a founding
member of the League of Professional System
Administrators.
Property of SNIA
40