Transcript Data Deduplication and Tivoli Storage Manager

Chicago Tivoli Storage Manager Users Group Meeting April 2010
Chicago Tivoli Storage Manager
Users Group Meeting
April 14, 2010
© 2010 IBM Corporation
© 2009 IBM Corporation
© 2010 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Introductions 10:00 - 10:15
 TSM 6.2 update 10:15 – 11:30
 Customer 6.2 upgrade experience 11:30 – 12:00
 Lunch 12:00 – 13:00
 FastBack and TSM 13:00 – 14:00
 Break 14:00 – 14:15
 STORServer 14:15 – 15:00
– FastBack offering
– STORServer reporter
– Live data gathering
– Data mining
– Capacity planning
– Troubleshooting
2
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM Version 6 Server Database
- Getting to the New Database
Chicago Tivoli Storage Manager Users Group Meeting
© 2010 IBM Corporation
© 2009 IBM Corporation
© 2010 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
4
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
5
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM 6.1 Deployments Are Growing
 TSM 6.1 GA’d March 2009 with significant new code and
functionality
 Early input from technical sales teams, support teams,
and customers suggests:
– TSM 6.1 adoption is growing, but slightly less than the 5.5 rate
– TSM 6.1 field reported problems are in line with prior releases
– Customers are hungry for more information about the new
database
• what’s new
• how to configure it
6
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Critical Issues Fixed in 6.1.2.1
 As of November 10, 2009, current fixes address key
known issues and continue to drive greater stability
– We encourage all customers to install maintenance level 6.1.2.1
• Most current maintenance available as of this date
• Available Now
 We are excited about TSM 6.1’s enhancements, and
remain focused on making our customer base and sales
teams successful
7
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
8
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM V6 Storage Components
Active Log
TSM Server
TSM DB
Log Mirror
(optional)
Archive Log
TSM STGPools (disk, tape)
9
Getting to the New DB2 Database © 2010 IBM Corporation
Failover
Archive Log
(optional)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The TSM Database
 Implemented via DB2
 Specified as a set of directories
– DB2 spreads the database across all directories
• Tries to maintain equal distribution across the directories
 Be generous in size estimates – plan for growth
– If you under allocate, DB2 may need to reorganize the database
• Done transparently, but time consuming
– To add space, you can either
• Add directories
• Make existing directories bigger
– Suggest you start with many smaller directories and make them
bigger as necessary
10
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Size Estimate for V6 Database
 Assume 600-1000 bytes per object stored
 Deduplication adds 250 bytes per extent per storage pool
– Number of extents is approximately
((( Average file size + 256KB ) / 256KB ) + 1) * (Number of Files)
• If average size is 2 MB, this would be:
( 2,097,152 + 262,144 ) / 262,144 = 9
( 9 + 1 ) * ( Number of Files )
– Calculations based on:
• Average extent (“chunk”) size is 256KB
• In addition to “data” extents, each file has 1 “metadata” extent
• Files between 2KB and 50KB will have 2 extents
– Difficult to determine “average” file size across storage pool
11
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Possible Database Layout – DB2 Managed
 Assume you estimate 200GB for disk space
 4 LUNs, each 50GB, assigned from Disk Array
 Each LUN is assigned its own Volume Group on host
 Each Volume Group has one File System
 DB2 will use separate I/O threads for each directory
LUN 1
VG1
/db1
50GB
12
LUN 2
VG2
/db2
50GB
LUN 3
VG3
/db3
50GB
Getting to the New DB2 Database © 2010 IBM Corporation
LUN 4
VG4
/db4
50GB
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Extending the DB – Option 1
 Create 2 new 50GB LUNs
 Assign to Volume Group and create File System
 Assign to TSM
LUN 1
VG1
/db1
50GB
LUN 2
VG2
/db2
50GB
LUN 3
VG3
/db3
50GB
LUN 4
VG4
/db4
50GB
LUN 5
VG5
/db5
50GB
LUN 6
VG6
/db6
50GB
 Caution! – DB2 will perform online reorganization
LUN 1
VG1
/db1
50GB
13
LUN 2
VG2
/db2
50GB
LUN 3
VG3
/db3
50GB
LUN 4
VG4
/db4
50GB
LUN 5
VG5
/db5
50GB
Getting to the New DB2 Database © 2010 IBM Corporation
LUN 6
VG6
/db6
50GB
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Extending the DB – Option 2
 Create 4 new 25GB LUNs
 Extend each file system by 25GB
– Can also be done on Windows via “Disk Management”
 No need to assign to TSM
 No need for DB reorganization
LUN 1
LUN 5
VG1
/db1
75GB
14
LUN 2
LUN 6
VG2
/db2
75GB
LUN 3
LUN 7
VG3
/db3
75GB
LUN 4
LUN 8
VG4
/db4
75GB
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Possible Database Layout – Array Managed
 Let the DS8K or other Disk Array manage the LUNs
 Assign 1 single 200GB LUN to the host
 One Volume Group and one File System
 DB2 will use separate I/O threads for each directory
Disk 1
50GB
Disk 2
50GB
Disk 3
50GB
Disk 4
50GB
Disk Array
15
LUN 1
200GB
Host
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
DB2 Usage of Array Managed Storage
 DB2 has parameter to control parallel I/O
– db2set –i <instance> db2_parallel_io
 TSM sets this to “*”
– Tells DB2 to assume this directory can handle multiple requests
– DB2 default is to use 6 I/O threads
• If your hardware can handle 12 concurrent I/O requests, then
– Login as instance user ID, and issue
– db2set db2_parallel_io=12
• You may need to restart the TSM server for this to take effect
16
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The TSM Log Directories
 Log space is more complicated than with TSM V5
– Required Log Directories
• Active Log Directory
– Contains all currently active transactions
• Archive Log Directory
– Contains all transactions required for DB restore to last backup point
– Optional Log Directories
• Active Log Mirror Directory
– Mirror copy of the active log
• Failover Archive Log Directory
– Failover directory for archive log
 All log directories should be tuned for sequential I/O
– No log directory should reside on the same disk or LUN as any
other log or database directory
17
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Log Size Estimates
 Active Log Directory
– Must be large enough to hold all active transactions
– 2GB minimum, but start with at least twice size of V5 log
• If the log fills up, the server halts – don’t underestimate
• You can reduce log size with server restart if you over allocate
– This is the highest priority of any log directory
• Should be on its own dedicated LUN or disk
 Archive Log Directory
– Must be large enough to hold all log files generated in 2 days
• Assuming you back up the database daily
• DB Backup removes log files from archive directory
– Performance not as critical
18
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Log Size Estimates
 Mirror Log (Active Log Mirror) Directory
– Same size as active log
– Same performance requirements as active log
 Failover Archive Log Directory
– Large secondary storage in case archive log directory fills
• Also used if archive log directory unavailable
– Can be a directory in a shared file system (NFS/CIFS)
– If this directory gets used frequently, then archive log too small
– Tune archive log to avoid using failover
19
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Log Size Examples
 This techdoc has numerous examples for log sizing:
– 1389352: Tivoli Storage Manager V6.1 server might crash if log is
not sized appropriately
• http://www-01.ibm.com/support/docview.wss?uid=swg21389352
20
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Preparation - Estimating Disk Requirements
Item
Type
Same system
Media
Same system
Network
New system
Media
New system
Network
Active Log
Disk
2GB (Min)
2GB (Min)
2GB (Min)
2GB (Min)
Log Mirror
Disk
Log Size
Log Size
Log Size
Log Size
Archive Log (1)
Disk
Log Size +
Log Size +
Log Size +
Log Size +
V5 DB
Disk
Current DB
Current DB
0
0
V5 Rcvylog
Disk
Current Log
Current Log
0
0
DB2 DB (2)
Disk
DB Util%
+ 50%
DB Util%
+ 50%
DB Util%
+ 50%
DB Util%
+ 50%
DB Backup (2)
Seq Media
DB Util%
DB Util%
DB Util%
DB Util%
Extract(2)
Seq Media
DB Util%
0
DB Util%
0
Total Disk
Disk
Total Seq
Seq Media
Note 1: Archive log is a function of daily activity
Note 2: V6 DB, DBB, and Extract are a function of current DB utilization
21
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Preparation – Sample for 100GB database 80% Utilized
Same system
Media
Same system
Network
New system
Media
New system
Network
Disk
32GB
32GB
32GB
32GB
Log Mirror
Disk
0
0
0
0
Archive Log(1)
Disk
100GB
100GB
100GB
100GB
V5 DB
Disk
100GB
100GB
0
0
V5 Rcvylog
Disk
13GB
13GB
0
0
DB2 DB
Disk
145GB
145GB
145GB
145GB
DB Backup
Seq Media
200GB
200GB
120GB
120GB
Extract
Seq Media
80GB
0
80GB
0
Item
Type
Active Log
Total Disk
Disk
390GB
390GB
277GB
277GB
Total Seq
Seq Media
280GB
200GB
200GB
120GB
Note 1: Archive log consumption changes in V6.1.2
22
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
23
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Memory Requirements in V5
 DB buffer pool largest memory consumer
– Contained within dsmserv/dsmsvc process
– TSM’s buffer pool not always efficient
 Linux and Unix
– 1-2G is reasonable
 Windows
– 1G is reasonable
– Problems with non-paged pool memory in Windows 32-bit
24
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Memory Requirements in V6
 DB buffer pool moved outside dsmserv process
– Managed by DB2 (db2sysc/db2syscs)
 8GB RAM Recommended
– DB2 buffer pool is larger
– DB2 more efficient buffer pool management
– DB pages are larger
 IPC facilities used for dsmserv/db2sysc communication
– Make sure you have high system limits on
• Shared Memory regions
• Message Queues
25
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
26
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
CPU Requirements for V6
 CPU requirements slightly higher than V5
 External Functions requiring more CPU include
– Deduplication
• SHA-1 and MD5 cryptographic functions highly CPU intensive
– Multi-process Expiration
• Trade-off between time and CPU
 Internal Functions requiring more CPU include
– DB reorganization
– DB runstats
• Used to optimize database lookups
27
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
28
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The Instance Model (V5 and V6)
 An Instance is everything required to run a server
– Database files
– Log files
– Configuration files (dsmserv.opt, volume history, etc.)
– Storage pool volumes
 Each instance requires it own directory
– dsmserv.dsk is always located in current working directory
– Database and log files usually stored separately
 You can have more than one instance per system
– Each instance is separate from every other instance
29
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The Instance Model (V6) – Unix
 On Unix, each TSM instance requires a unique user ID
– Doesn’t need to be dedicated to TSM use, but not a bad idea
• Cannot be root
– Each instance contains 1 TSM database
 If you require more than 1 server on a system, you need
more than 1 unique instance user ID
– TSM Instance name = db2 instance name = instance user ID
30
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The Instance user ID and root – Unix
 Instance user ID controls database access
– Primary group of instance user ID is administrative group of DB
– Only members of this group can start/stop the database manager
• Including root – root must be a member of this group, too
# id tsmuser
uid=203(tsmuser) gid=1(staff) groups=0(system),2(bin)
# id root
uid=0(root) gid=0(system) groups=2(bin),3(sys),7(security),8(cron)
As tsmuser:
$ db2 get dbm cfg | grep SYSADM
SYSADM group name
(SYSADM_GROUP) = STAFF
Note: root is not authorized in this example
31
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The Instance Model (V5 and V6) – Windows
 Windows has had an instance model for years
– dsmserv –k <key>
• Default is “server1”
• HKLM\Software\IBM\ADSM\CurrentVersion\Server\<key>
– Model not changed for V6
 TSM instances can share a user ID
– TSM Server Key = db2 instance name
• For example, “Server1”
• Instance user can be any user ID
32
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM V6 Storage Components
Database
Manager
(tsmuser)
DB
Requests
TSM DB
Active Log
Log Mirror
(optional)
TSM Server
(tsmuser, root, or
Administrator)
Archive Log
TSM STGPools (disk, tape)
Configuration Files
dsmserv.opt, trace,
devconfig, volhist
33
Getting to the New DB2 Database © 2010 IBM Corporation
Failover
Archive Log
(optional)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
A Look at the Processes – Unix
Server
Database
Manager
IPC
dsmserv
Shared Mem
Msg Queues
db2sysc
tsmuser 365186 209986
0 11:13:36
-
0:00 db2acd 0
tsmuser 246764 209986
0 11:13:35
-
0:25 db2sysc 0
root
0 11:13:34
pts/0
328786 107714
0:03 dsmserv
 db2sysc is the main database manager process
 db2acd is the DB2 health monitor
 Processes communicate via IPC
34
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
A Look at the Service List – Windows
35
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
A Look at the Service List – Windows
36
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
37
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Preparing for the Upgrade
 Read the DB Upgrade Utility README
 Obtain the DB Upgrade Utility
– Separate from TSM V6 Product
– Always get the latest available from the FTP site
 Obtain the TSM V6 DVDs
– Back up V5 database before installing V6
 References:
– DB Server Upgrade Guide (SC23-9554)
– Storage Technical Exchange Website:
– http://www-01.ibm.com/software/sysmgmt/products/support/supp_tech_exch.html
38
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM V6 Restrictions
 Cannot switch platforms using Upgrade Utility
– Following architecture upgrades supported
•
•
•
•
Windows x86 to Windows x64
Windows IA64 to Windows x64
HP PA-RISC to HP IA64
32-bit to 64-bit on same platform with same endianness
– For example, AIX 32-bit to AIX 64-bit
 Cannot merge multiple V5 databases during upgrade
 Cannot alter the underlying DB2 settings
– Pre-configured by TSM during installation and format
39
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
40
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
The Log Directories (Revisited)
 Active Log Directory
– Linear (non-circular) fixed-size log in a single directory
• Performance and availability are very important
– Broken up into 512MB files
 Archive Log Directory
– Contains archives of active log files for database restore
 Mirror Log Directory
– Mirror of the active log directory – same size
 Failover Archive Log Directory
– To handle overflow of archive log directory
41
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
How the logs flow
 Transaction starts and stops are written to active log
 Once an active log file is full, it is immediately copied to
an archive log directory
– If the archive log directory is not writeable, it is copied to the
failover archive log
– If the failover archive log is not writeable, it is not copied
 When an active log file has no more active transactions
within it, it is eligible for deletion
– Cannot be deleted until it is archived
42
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
How the logs flow
Full
Txn 1
Full
Full
Current
S0000100.LOG
S0000101.LOG
S0000102.LOG
Txn 2
Txn 3
Txn 4
Active
S0000099.LOG
…
S0000098.LOG
S0000099.LOG
S0000100.LOG
Archive (Now Full)
Failover Archive
S0000101.LOG
43
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Keeping the logs flowing
 To keep the active log from filling:
– Make sure the active log is large enough to contain active txns
• Watch out for slow clients pinning the log
• In TSM V6, there is much more log activity than in TSM V5
• If your active log is too small, it will fill and the server will halt
– Make sure the archive log directories have space
 To keep the archive log directories from filling:
– Perform regular FULL DB backups
• Only FULL DB backups will clear the archive log directories
 It may take some time, but if ANY of the log directories
becomes full, the server may halt
– The reason for the halt will be out of active log space
44
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Database Maintenance Tasks
 Table Reorganization
– Performed when CPU and I/O activity low
– DB2 optimizes database tables for efficiency
– Generates a lot of active log records
• Can interfere with long-running transactions
ANR0293I Reorganization of table <table_name> started.
ANR0294I Reorganization of table <table_name> ended.
 Statistics Updates (runstats)
– Used by DB2 to optimize TSM’s SELECT statements to the DB
• Improves DB2’s ability to use indices to avoid table scans
ANR0136I Table updating statistics performed successfully for n of n.
45
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Agenda
 Current Field Status
 Storage requirements for the new database and logs
 Memory requirements
 CPU requirements
 User ID requirements
 Upgrade Considerations
 Examining database and log operations
 Backing up the database
46
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Configuring V6 for DB Backup
 V6 database backup uses the TSM client API
– API installed automatically with V6 server
– Uses a special nodename “$$_TSMDBMGR_$$” for backup
• Password MUST be TSMDBMGR
• This node is hidden and can perform only DB backup and restore
– Note: Be careful when canceling sessions. It is possible to cancel
the API session doing the DB Backup
 V6 database backup and restore require volumehistory
and devconfig files
 If possible, use either instance config or upgrade wizard
– Set the password in TSM.PWD or registry entry
– Sets up dsm.opt, dsm.sys, and DB2 parameters
47
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Processing Flow of DB Backup
1
TSM
DB2
Database
2
TSM
Server
3
1 Initiate DB Backup
2 Intercept Inbound Session from DB2
3 Stream DB Backup to Sequential DataStream
Sequential DataStream
(Seq Disk or Tape)
4
Volhistory
4 Volume history file is written
48
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
DB Backup Types
 Full Backup
– Backs up entire database and all archive logs
• If using FILE device class, stores archive logs on separate volume
• If using TAPE device class, appends logs to DB backup volume
– Prunes the archive log directories
 Incremental Backup
– Backs up all archive logs since last DB backup
• Does not back up the database itself
– Does not prune the archive log directories
49
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Differences in V6 Incremental Backup
 Different than V5 incremental DB backup
 For V6 DB restore, need
– FULL backup,plus
– LAST incremental backup
Full
Inc
Inc
Inc
Inc
Inc
Inc
Full
V5
Sun
50
Mon
Tue
Wed
Thu
r
Fri
Getting to the New DB2 Database © 2010 IBM Corporation
Sat
Sun
V6
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
IMPORTANT FLASHES
As of November 10th, 2009
© 2010 IBM Corporation
© 2009 IBM Corporation
© 2010 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Important Flashes
 1408717: The Tivoli Storage Manager patch 6.1.2.1 is
available and recommended for use by all customers
– http://www-01.ibm.com/support/docview.wss?uid=swg21408717
 1389352: Tivoli Storage Manager V6.1 server might
crash if log is not sized appropriately
– http://www-01.ibm.com/support/docview.wss?uid=swg21389352
– Contains important log sizing examples
 1406035: Active log fills when a transaction runs for a
long time
– http://www-01.ibm.com/support/docview.wss?uid=swg21406035
52
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Important Flashes
 1406032: Tivoli Storage Manager V6.1 server might
crash in DB2 operations
– http://www-01.ibm.com/support/docview.wss?uid=swg21406032
 1394341: Failure during client backup or archive
operation when storage destination is DISK storage pool
with CACHE=YES enabled
– http://www-01.ibm.com/support/docview.wss?uid=swg21394341
 1408547: MOVE DATA and MOVE NODEDATA with
RECONSTRUCT=NO might cause data integrity issues
– http://www-01.ibm.com/support/docview.wss?uid=swg21408547
53
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
BONUS SECTION
Example Upgrade Scenario
© 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Upgrade Example
 Library Sharing Environment
– 2 OS Instances
– Library Manager with a shared tape library
– 2 Library Clients
 LANFree
– 2 Storage Agents using 1 Library Client
 Preparation
– Upgrade Servers and Storage Agents to minimum supported
levels to work with New TSM
– Install / Upgrade OS and Hardware as appropriate
– Update SAN Zoning to include new paths
55
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example: Starting Configuration
OS2
OS1
LIBR CLIENT 1
LIBR MGR
Server to Server
V5
LIBR CLIENT 2
Server to Server
V5
V5
SAN Paths
Storage Agents
Shared Library
56
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example Upgrade Scenario – Library Manager
 Move Libr Mgr to new OS instance (OS3)
– Update SAN Zoning
– Update Libr Mgr Paths
– Update Libr Client connections
– Validate configuration / paths / connectivity
 Upgrade Libr Clients and Storage Agents
 Upgrade Libr Mgr to V5.5.2
– Validate configuration / paths / connectivity
 Upgrade Libr Mgr to New TSM in place
– Small DB, should be fairly easy and fast
– Validate configuration / paths / connectivity
!! This is not the only way to upgrade !!
57
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example: Library Manager Upgraded
OS1
OS2
OS3
LIBR CLIENT 1
LIBR CLIENT 2
LIBR MGR
Server to Server
Server to Server
V5
New
Paths
V5
Upgraded
Storage Agents
Shared Library
58
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example Upgrade Scenario – Library Client 1
1. Build Libr Client 1 Instance on OS3
2. Upgrade Libr Client 1 using Media Method
– Shutdown Libr Client 1
– Extract Database
– Start Libr Client 1
3. Create Libr Client 3 on OS3
– Insert DB into Libr Client 3 (from Libr Client1 extract)
4. At this point Libr Client 3 is identical to Libr Client1
– Rename new Libr Client1 to Libr Client3
!! This is not the only way to upgrade !!
59
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example Upgrade Scenario – Library Client 1 …
5. Update Connectivity
– Add Libr Mgr Paths for Libr Client 3
– Update Libr Client 3 connections
– Validate configuration / paths / connectivity
6. Define connectivity between Libr Client 1 and Libr Client 3
7. Update B/A client connectivity
– Either update clients or update TSM Server and host OS
8. Export/Import from Libr Client 1 to Libr Client 3 using date/time
9. Shutdown Libr Client 1
!! This is not the only way to upgrade !!
60
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example: Library Client 1 Upgraded
OS2
OS3
LIBR CLIENT 3
LIBR MGR
Server to Server
New
LIBR CLIENT 2
Server to Server
V5
New
Paths
Upgraded
Storage Agents
Shared Library
61
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example Upgrade Scenario – Library Client 2
1. Install Upgrade utility on OS2
2. Shutdown Libr Client 2
3. Install New TSM on OS2
4. Create Libr Client 2 DB instance
5. Upgrade Libr Client 2 in place using network method
6. Start Libr Client 2
7. Validate configuration / paths / connectivity
– Libr Mgr
– Storage Agents
– B/A Clients
!! This is not the only way to upgrade !!
62
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Example: Library Client 2 Upgraded
OS2
OS3
LIBR CLIENT 3
LIBR MGR
Server to Server
New
LIBR CLIENT 2
Server to Server
New
New
Paths
Upgraded
Storage Agents
Shared Library
63
Getting to the New DB2 Database © 2010 IBM Corporation
© 2009 IBM Corporation
Tivoli Storage Manager
TSM V6.2 Technical Overview
David Larimer
Senior IT Specialist for Tivoli Storage
[email protected]
© 2010 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Functions / Topics
 Windows Backup/Archive Client Support for GPFS 3.3
 Higher Limits on TXNBYTELIMIT and MOVESIZETHRESH parameters
 Windows Passthru Driver
 Automatic Deployment for Windows Clients
 TSM for ERP: Improve TSM Handling of Very Large Objects
 Microsoft Hyper-V full Guest Backup Using Volume Shadow Copy Services (VSS)
 Simultaneous Write During Storage Pool Migration
 Concurrent Access to Centera Volumes
 Auto-discovery of New VMware Guests
 VCB Gen II - VMware vStorage for Data Protection API integration
 Security: In Flight Data Encryption Using SSL
 Client-side Data Deduplication
 Incremental Backup of Windows System State
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Backup/Archive Client Support for GPFS 3.3
 TSM already supports GPFS on AIX and Linux
 NEW: TSM Windows support for GPFS:
– File system recognition
• for Q FILESPACE
• for client GUI and client command line
– Backup, Restore, Archive, Retrieve
– Backup and Recovery of security permissions
– Backup and Recovery of GPFS storage pool information
– Support only for Windows Server 2008 x86-64 (no 32 bit
support)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows B/A Client Support for GPFS 3.3
 TSM Windows support for GPFS (cont.):
– LAN-Free data movement
– Backupset
– Adaptive Sub-file backup
– Support for non-Administrator credentialed users
• must be member of Backup-Operators group
– Cross file system restore
• i.e. backup on GPFS and restore to NTFS file system and vice-versa
 Not Supported:
– Image Backup, Journal Based Backup, Open File Support, HSM,
Cluster Failover, Cross platform restore, Multiuser support, Case
Sensitivity
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Limits on TXNBYTELIMIT and MOVESIZETHRESH
 Limits increased to 32 GB
 TXNBYTELIMIT set in dsm.sys (unix) or dsm.opt (Windows)
 MOVESIZETHRESH set in dsmserv.opt or with SETOPT command
 Can specify TXNB units (default is Kbytes):
–
TXNB 300M (option file)
–
-txnb=32G
(command line)
 MOVESIZETHRESH is specified in Mbytes
 Use larger settings for large backups with fast tape drives
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 Why make this change:
– Windows Hardware Qualification Lab (WHQL)
• Some companies have security policies which require certified
drivers
– Install of TSM device driver generates pop-ups
• Driver cannot be installed silently
– TSM device driver cannot be certified with WHQL
• Supports non-IBM devices – cannot submit a request with any
non-IBM devices
• Uses TSM defined IOCTLs that are not known to Microsoft
• Does not support all standard IOCTLs
• Supports many old devices – would be impossible to test all
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 Native device drivers –
– A device driver that supports all standard IOCTLs for a
particular device type
– TSM device driver is not a native device driver
– IBM device driver for IBM devices like LTO & 3592 is a
native device driver
– Microsoft driver or operating system device driver is a
native driver
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 TSM Server communicates with devices via SCSI passthru interface in
device driver
– IOCTL_SCSI_PASS_THROUGH
– IOCTL_SCSI_PASS_THROUGH_DIRECT
 TSM device driver now supports these SCSI Passthru IOCTLs
– Native drivers also support these
 Users can use either the native device driver or the TSM device driver
 TSM Server communicates the same way with devices via both drivers
 When upgrade to TSM 6.2, customers can switch to the native driver
– No changes to the existing device definition necessary
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 Considerations:
– Users will need to find the native driver for each device
• Either use the Microsoft built-in driver
• Or get the driver from the device vendor
– Some devices no longer supported by Windows 2003 or Windows 2008
• TSM will continue support of TSM Device Driver
• Driver selection can be done a per device basis
 Externals Considerations:
– New parameter in DEFINE PATH command: GENERICTAPE=YES|NO
(Defaults to No)
• Allows users to define a GENERICTAPE type device for either supported or nonsupported drives
• Users can define an unsupported drive as GENERICTAPE
– If that drive becomes a supported device, users can continue to use it as
GENERICTAPE
• For supported devices, TSM format (such as LTO) is recommended
– Use GENERICTAPE=Yes only when the device is not supported by TSM
• The native device driver is required for GENERICTAPE
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 Externals Considerations:
GENERICTAPE
Value
Device
Drive DevType
Yes
Supported
GENERICTAPE plus a Warning
No
Supported Non-IBM Device
4MM, 8MM, DLT, DTF, ECART,
LTO, or QIC
No
Supported IBM Device
3570, 3590, 3592, LTO
Yes
Unsupported Device
GENERICTAPE
No
Unsupported Device
Invalid with New Error Message
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Windows Passthru Driver
 Externals Considerations:
– Device Configuration File includes GENERICTAPE parameter in
DEFINE PATH commands
– TSMDLST.EXE will change to show the device type for supported
drives regardless of which driver is in control.
•
•
Unsupported devices will continue to show as GENERICTAPE
TSMDLST output will include a column for Driver type: TSM, IBM, NATIVE, or
N/A
– TSM not able to reset a drive for failover support when using a native
device driver
•
For cases where the TSM Library Manager is on Windows and LUN reset is
required, perform a workaround or use the TSM device driver for at least one
device
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
 Addresses need to centrally manage and perform the
distribution and upgrade of TSM client code
 For TSM 6.2:
– TSM administrator able to configure & schedule automatic client
maintenance
– For Windows Backup/Archive client only
– For upgrade from V5.4.X.X (or higher) to V6.2.0.0 (or higher)
– It is the goal to provide upgrade for other platforms and releases in
the future
– Requires a TSM 6.2 server
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
 Capabilities from the Admin Center for TSM Admins:
– Discover client maintenance levels on the TSM FTP site
– Download needed packages and store them on the TSM
server
– Manage packages stored on the TSM server (retention, etc.)
– Select a maintenance level and distribute to a list of clients
• The code will then be distributed based on a predefined schedule
– Review the client distribution status
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
 Not Supported in TSM 6.2:
– Distribution of other TSM components such as Storage
Agent, Data Protection modules, HSM, etc.
– Distributions on non-Windows clients
– Distribution for downgrade or rollback (e.g. from 6.2.1.1 to
6.2.1.0)
– Ability for clients to auto-discover a new level and
upgrade w/o admin action
– Distribution for initial install
– Distribution when the client scheduler is not running
– Use of Admin Center versions prior to 6.2
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
 How it works:
– Special objects created on TSM server with name prefix of
IBM_CLIENT_DEPLOY
– Admin Center provides several wizards to configure and perform
distribution
– Device Class: FILE type that identifies the location where maintenance
packages will be imported from
• Admin Center automatically moves packages to this location
• Or packages can be manually placed in this location
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
 How it works:
– Node: special node name will control the Archive packages that are
created as a result of the Import of packages
• A separate node will be create for each platform (only
IBM_CLIENT_DEPLOY_WIN in this release)
– Domain: will hold all the nodes and schedules that are created to
support distribution
• Admin Center will create the policy structure (mgmt classes, etc.) for this
domain
– Storage Pool: Admin Center will create a dedicated FILE type storage
pool to hold the imported packages
• Alternatively, an administrator can use an existing FILE or DISK type storage
pool
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Automatic Deployment for Windows Clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM for ERP: Improve TSM Handling of Very Large Objects
 The challenge TSM backups see are:
– Customer db are growing to multiple terabytes
– Difficult to cancel a running backup, as it takes a long time to “clean up” the cancelled
activity
– The TSM Recovery Log can get pinned while processing very large objects
 The solution is to:
– Break very large objects into smaller pieces
– Perform TSM db commits as each of these pieces is successfully backed up
 TSM for ERP with DB2 segments & manages db backup objects
– Does not apply to TSM for ERP with Oracle databases
– Done by the TSM for ERP module and is transparent to the TSM server and to DB2
– TSM for ERP BackOM and Administrative Assistant are changed to report logical
backup sizes (as opposed to reporting segment sizes)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
TSM for ERP: Improve TSM Handling of Very Large Objects
 A new parameter is introduced for the TSM for ERP profile (initSID.utl)
SEGMENTSIZE <size> [GB | TB]
– All Backup objects broken into units equal to SEGMENTSIZE
•
Doesn’t apply to Log archives
– A TSM db transaction commit issued after each segment is successfully transmitted
•
Frees resources on the TSM server (particularly Recovery Log entries)
 TSM for ERP uses a suffix on the object name
– Helps keep track of the all the segments for a given db backup on the TSM server:
<DB2 instance>.<db alias>.<type>.<partition num>.<DB2 backup id>.<session num>:<segment num>
– Works well with DB2’s object delete mechanism which is based on a filter:
<DB2 instance>.<database alias>.<backup type>.<partition number>.<DB2 backup timestamp>.*
 TSM for ERP uses a special zero-length object to store metadata about the set of backup segments:
<DB2 instance>.<db alias>.<type>.<partition num>.<DB2 backup id>.<session num>:C<last
segment num>
– Written after all segments are successfully transmitted
– Used to determine # of segments to ensure the integrity of the restore
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 Windows Server 2008 Hyper-V
– Hypervisor implementation from Microsoft –
– Follow-on to Microsoft Virtual Server 2005
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 Windows Hyper-V requirements:
–
–
–
–
64-bit processor (no 32-bit or Itanium versions) (Guests can be 32-bit)
Dual-Core (or more) processor
Hardware-assisted virtualization (Intel VT, or AMD-V)
Hardware enforced Data Execution Prevention (Intel XD bit or AMD NX bit)
 Windows Server 2008 Core installation:
– Server Core is an install with limited subset of functions – can create slim install
– Can include Hyper-V function
•
Does not include graphical interface (command line administration only)
– TSM command line (no GUI) supports Server Core implementations (including Hyper-V functions)
 Windows Volume Shadowcopy Service (VSS)
– VSS creates snapshots of the running Hyper-V guest virtual machines
– TSM Hyper-V support is built on top of the existing TSM VSS support
– VSS snapshot awareness is propagated to apps (i.e. Exchange, SQL…) within the guest virtual
machine
•
So even the DB’s inside of the virtual guest are backed up properly with VSS
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 TSM Hyper-V Guest Backup and Restore
– Only full guest virtual machine support
•
No visibility to individual files within the guest
– Guest virtual machine is basically a single entity
•
•
Large .vhd file with a few small supporting files
Can be backed up from the host operating system if the guest virtual machine is shut down
– VSS is used to create a snapshot of the running guest virtual machine
•
•
•
TSM sends commands to a Hyper-V VSS Writer interface
This Writer propagates the requests to all internal VSS Writers (Exchange, SQL-Server, etc.)
– TSM does not interact with the internal Writers directly
This ensures the integrity of the guest virtual machine and all internal VSS applications
– Cluster Shared Volumes often used with Hyper-V to support live movement of a guest
virtual machine from one physical machine to another
•
•
TSM’s VSS supports allows Hyper-V guest virtual machine backup where Cluster Shared Volumes are involved
TSM B/A client supports the backup and restore of Hyper-V guest virtual machines
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 TSM Command Line Client:
– New option on Query VM, Backup VM and Restore VM commants:
•
VMBACKUPTYPE=HYPERVFULL
 TSM GUI Client:
– When GUI client detects that it is running on a Hyper-V server:
•
•
•
GUI will display a list of Hyper-V guest virtual machines that can be backed up or restored
TSM will backup or restore all files associated with a guest virtual machine
TSM server uses the file grouping function to maintain integrity of the guest files
– Hyper-V guest virtual machine backups can create versions
– Multiple guest virtual machines can be restored
•
All the guest virtual machines for given host are stored under one Node name
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 Backup
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Microsoft Hyper-V Full Guest Virtual Machine Backup Using
Volume Shadow Copy Services (VSS)
 Restore
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Simultaneous Write During Storage Pool Migration
 Simultaneous Storage Pool Backup at client backup time has been available with
TSM since V5.1
– Can slow backup and it can significantly increase tape drive usage
 Manual Migration and Storage Pool Backup need to be done in separate windows
of time
– These windows are getting smaller
– It is more difficult to complete these functions separately
 These same issues apply for simultaneous write to Active Data Pools
 New feature allows simultaneous Migration & Storage Pool Backup and/or Active
Data Pool Copy
– Admins can specify up to 3 Copy Pools
– Copies are incremental (no change here)
– The source primary pool can be random or sequential, disk or tape
•
Must use the NATIVE or NONBLOCK format
– Simultaneous write will not support other data movements
•
Reclamation, Move Data, Move Nodedata, etc.
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Simultaneous Write During Storage Pool Migration
 Implementation:
– Existing COPYSTGPOOLS & ACTIVEDATAPOOLS parameters of the DEFINE STGPOOL
and UPDATE STGPOOL commands are used
• STEP 1: Specify a storage pool value (target pool) on NEXTPOOL parameter for the migration source
pool (not new)
• STEP 2: For the NEXTPOOL specify the COPYSTGPOOLS and/or ACTIVEDATAPOOLS parameters
(up to three total) for the migration source pool
• STEP 3: Specify a value for AUTOCOPY parameter (this is a new parameter) on target pool:
– NONE – disable simultaneous write altogether
– CLIENT – perform simultaneous write only on client backup operations
> Also applies to IMPORT for Copy Storage Pools only
> This is the default
– MIGRATION – perform simultaneous write only on migration operations
– ALL – perform simultaneous write during both client backup operations
> Including IMPORT for Copy Pools
> And during migration operations
– The COPYCONTINUE parameter is not used for simultaneous copy during migration
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Concurrent Access to Centera volumes
 Goal: Provide same concurrent access as done for devclass=file
devices
 Internal changes only, no externals
 With concurrent access, you will have more sessions so consider
raising
– MAXNUMMP for clients accessing Centera
– MOUNTLIMIT on Centera device classes
– Note: The sum of all mount limit values for all device classes assigned to the
same Centera device should not exceed the maximum number of sessions
allowed by Centera.
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
VMWare Support Enhancements
 Auto discovery of New VMware guests
 VCB Gen II - VMware vStorage for Data Protection API
integration
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
vCenter Structure
Folder
DataCenter
Guest
Virtual
machine
Host
Physical machin
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Auto discovery of New VMware guests
 Uses VMware Infrastructure SDK (VI API)
 VMCHOST still specifies where to connect
– vCenter or ESX server
 ESX password encrypted in registry when using:
– GUI preferences editor
– Command “dsmc set password type=vcb ...”
 Note: Not encrypted if coded directly in options file
 TSM connects to VMCHOST and obtains list of all VM guests including
– Host ESX server
– Owning folder
– OS type
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Auto discovery of New VMware guests
 VCB file level backup requires the TSM admin to define a nodename for
the guest and setup proxy node authority
– REGister NOde VMGUEST1 password
– REGister Node PROXYNODE password
– GRant PROXynode TArget=VMGUEST1 AGent=PROXYNODE
 Backup vm -vmbackuptype=fullvm
– Backup occurs as all backups use the same Nodename
 Backup vm -vmbackuptype=file (Windows only)
– Backup of VM fails if Nodename and Proxy have not been defined
ANS1662I Agent Node: 'backupproxy1' Target Node: 'shuffle'
ANS1532E Proxy Rejected: Proxy authority has not been granted to this node.
ANS4150E Incremental backup of Virtual Machine 'shuffle' failed with RC 5722
– These errors appear in server activity log
• Visible to TSM Admin without searching client error log
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
DOMAIN.backuptype option
 DOMAIN defines guests to be processed for backups
– TSM uses the domain when processing a backup vm command
without specifying which virtual machines to process
 Only one DOMAIN statement for each backup type
– DOMAIN.VMFULL
• Creates list of VMs eligible for full VM backups
– DOMAIN.VMFILE
• Creates list of VMs eligible for file-leve backups
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
DOMAIN Syntax
 ALL-VM
– Selects all virtual machine guests found on vCenter or ESX
– Not valid for DOMAIN.VMFILE
 ALL-WINDOWS
– Selects all Windows guests
 VMHOST=esx1.ibm.com,esx2.ibm.com
– Selects all guests on an ESX host
 VMFOLDER=foldername1,foldername2
– Selects all guests with a folder
 Use only one of the above
 VM=guest1,guest2
– Adds to list of guests
 -VM=guest3,guest4
– Excludes from list of guests
 Examples
– DOMAIN.VMFULL ALL-VM; -VM=test1,test2
– DOMAIN.VMFILE VMHOST=esx1.ibm.com;VM=guest99
– DOMAIN.VMFILE VMFOLDER=PRODUCTION
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Auto Discovery requirements
 Support
– ESX / ESXi – 3.5 and 4.0
– vCenter - vSphere 4 and VI 3.5
 Proxy Platforms
– Windows Server 2003 (32 bit and 64 bit)
– Windows Server 2003 R2 (32 bit and 64 bit)
– No IA-64bit support
– Windows Server 2008 (32 bit and 64 bit) – new !
– Windows Server 2008 R2 (64 bit only) – new !
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
VCB Gen II - VMware vStorage for Data Protection API Integration
 VMware Introduced a new interface
– vStorage for Data Protection API
 Replaces VCB for file level backups,
– No calls to VCBMOUNTER executable
– No need for VCB Framework installation
 No mountpoint exposed
– Uses a symbolic link in the global name space
– VMBACKDIR not required for filelevel backup
 No external changes
 vStorage API will detect the best data transfer path - SAN or LAN
 VCB still used for FULLVM backups
– vStorage and VCB Framework can coexist
– VCB might be withdrawn from market, but will still be available for use
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
VMware VCB
Abstract
VMware recently announced end of availability for VMware Consolidated Backup (VCB). Reference:
http://app.connect.vmware.com/e/es.aspx?s=524&e=12880125 for VMware's announcement.
recovery of VMware environments?
How does this affect TSM support for backup and
Solution
This information is provided in reference to VMware's "VMware Consolidated Backup - End of Availability Announcement" which was distributed by
VMware through their Partner Newsletter on 02/24/2010. In this statement VMware stated " ... with the release of the next vSphere platform, we will
continue to provide the binaries for VCB, but they will not be compatible with the next platform release. We will continue to provide support for VCB on
the current vSphere platform ...". Note the following in reference to TSM support for VMware environments:
Tivoli Storage Manager will continue to provide support for backup and recovery for the current VMware Virtual Infrastructure and VMware vSphere platforms.
Refer to the following table for details and software prerequisites based on your current TSM and VMware environments and current method for backup and
recovery.
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Security: In Flight Data Encryption Using SSL
 New for TSM V6.2
– Provide SSL security to multiple platforms
– Extend Certificate support to externally signed certificates
• Verisign, Thawte, etc.
• Or an internal Certificate Authority
 Two Supported Certificate types:
– Self-signed
• As used in TSM V5.5 and TSM V6.1
– Externally signed
• Either Well-Known Certificate Authority or not Well-Known Certificate
Authority
• Can create your own with various utilities
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Enhanced SSL Support for Deduplicated Data
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Security: In Flight Data Encryption Using SSL
Installation & Support
 V6.2 installation packages include all requirements
– GSKit is included in the distribution
– V7 or V8 – depending on platform
 Only Client to Server sessions are supported
 Not supported sessions:
– client to client
– web browser to client
– server to server
– stagent to server
– admin center to server
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Security: In Flight Data Encryption Using SSL
Server Implementation
 Customize server options for SSL
– Server Options
•
•
•
SSLTCPPORT
SSLTCPADMINPORT
SSL and non-SSL ports must be different
 Restart TSM server to generate the keyring file
 Issue SET SSLKEYRINGPW
– To create a useful password
 Obtain Certificate from CA
– Or generate your own
– Install ‘root certificate’ if not well-known CA
 Install Certificate
– Set as Default certificate
 Issue QUERY SSLKEYRINGPW
– To review
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Security: In Flight Data Encryption Using SSL
Client Implementation
 Client option file
– SSL YES
– Set TCPPORT to match server SSLTCPPORT
 For Well-Known Certificate Authority
– No further action (root certificates are installed with the TSM
client)
 For not Well-Known Certificate Authority
– Install the root certificate on the clients
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Comparison of TSM Data Reduction Methods
Client
compression
Incremental
forever
Subfile
backup
Server-side
deduplication
Client-side
deduplication
How data reduction is
achieved
Client
compresses files
Client only sends
changed files
Client only sends
changed subfiles
Server eliminates
redundant data
chunks
Client and server
eliminate
redundant data
chunks
Conserves network
bandwidth?
Yes
Yes
Yes
No
Yes
Data supported
Backup, archive,
HSM, API
Backup
Backup
(Windows only)
Backup, archive,
HSM, API
Backup, archive,
API
Scope of data reduction
Redundant data
within same file on
client node
Files that do not
change between
backups
Subfiles that do
not change
between backups
Redundant data
from any files in
storage pool
Redundant data
from any files in
storage pool
Avoids storing identical
files renamed, copied, or
relocated on client node?
No
No
No
Yes
Yes
Removes redundant data
for files from different
client nodes?
No
No
No
Yes
Yes
Available 6.1
Available 6.2
Available prior to V5
All of these data reduction methods conserve storage pool space
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication
Overall Goal:
 Reduce network traffic
–
Client-side/distributed deduplication
 While providing:
–
–
–
Optimization of the Deduplication processes
Security and data integrity
Activity reporting
Changes:
 Server, Client, and API* Updated
–
–
–
Transparent to basic operations
Logical Restrictions
New options / controls
 Client-side performs
–
–
Data fingerprinting (identification of data extents or chunks)
Data digest generation (hashing)
 Client sends hash and unique chunks to server
–
Chunks can be compressed before sending
*(API client side dedupe coming in 6.2.x)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Operation
B File 4 F
E
DeduplicationEnabled Disk
Storage Pool
Local Cache
1. Client creates chunks
2. Client and server identify which
chunks need to be sent
TSM 6.2 client
A
FileB1
C
D
F
TSM 6.2 API
E
4. Entire file can be
reconstructed during
Backup Stgpool
operation to nondeduplicated stg. pool
at a later time
Copy Storage Pool
Local Cache
(non-deduplicated)
3. Client sends
chunks and hashes
to server so that it
can represent object
in database. Client
saves newfound
hash values in local
cache.
File 1
File 4
File 2
File 3
hash
File 4
Index
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Prerequisites
 Client and server must be at version 6.2.0
 Client must have the client-side deduplication option enabled
 Server must enable the node for client-side deduplication
– DEDUP=CLIENTORSERVER parameter with REGISTER NODE or UPDATE NODE
 Storage pool destination for the data must have deduplication enabled
 File must not be excluded from client side deduplication processing
– By default all files are included
– See exclude.dedup Client option for details
 File must be larger than 2 KB
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client Side Overview
Design point
Comments
Source-side (client-side)
•Reduces network traffic by deduplicating data before transfer to the server
•Data can be sent from 6.2 Backup-Archive or API client to 6.2 server
In-band
•No post processing required after data is stored in deduplication-enabled
disk storage pool
Data compatibility with
server-side data
deduplication
•Client-side and server-side deduplication share data chunks in pool using a
unified chunk index in the TSM database
•Client-side and server-side use the same hashing algorithms/parameters
for fingerprinting and chunk identification
Optimization
•Client maintains a local cache of hash values
•Minimizes network chat and database lookups due to chunk-index
(deduplication-index) queries to the TSM server
Compatibility with client
compression
•Compressed and uncompressed chunks are interchangeable (only one
form is stored)
•Server expands (decompresses) data when it needs to be reconstructed,
such as for backup to a tape copy pool or for restore to a legacy client
Server and client
controls over
deduplication used
•Server enables each client node for client-side deduplication
•Client controls whether it actually uses client-side deduplication
•Client include/exclude options allow control of client-side deduplication at
the file level
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Logical Restrictions
 LAN-free
 Unix HSM
 Encryption
– TSM client-side
– Files from known encrypted file systems which are read in their raw
encrypted format
– SSL encryption is OK
 Sub-file backup
 API Buffer copy elimination
 Simultaneous Storage Pool Write
If any of these exist, Client-side dedup does not occur
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Options / Controls
 Server-side
– Register and Update Node commands include:
• DEDUPLICATION=SERVERONLY | CLIENTORSERVER
– Copygroup destination must be
• Stgpool with DEDUPLICATE=YES
 Client-side
– DEDUPLication YES | NO
– Include.dedup and exclude.dedup
– ENABLEDEDUPCAche YES | NO
– DEDUPCACHEPath directory_name
– DEDUPCACHESize 256 (1 to 2048 in MB)
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Include / Exclude Controls
 Include.dedup filter
 Exclude.dedup filter
 Objtype parameter
– Include and Exclude may also specify:
•
•
•
•
File
Image
SYSTEMObject
SYSTEMState
 Examples
–
–
–
–
include.dedup /Users/Administrator/Documents/.../*
include.dedup objtype=image E:
include.dedup objtype=systemobject ALL
include.dedup objtype=systemstate ALL
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication
Preferences Editor & Include / Exclude controls
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Optimization
 Local cache for hash index
– Client controlled size (around 200mb for 500gb of backup data)
– Cleared when integrity is questioned
– Uses system cache memory
 Local cache similar to JBB and SnapDiff implementations
 Additional session for digest queries to server
 CPU increase of 3x
 Elapsed time upto 2x shorter if large % of duplicate data found
 I/O and Memory are not significantly different
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication and Compression

New chunks can be compressed

Process:
–
–
–
–
Client chunks the data
Client compresses each chunk
Data is stored in compressed format
During a restore operation, compressed chunks sent back to client

Compressed chunks shared across compressed/not compressed objects

Server will decompress on way back to lower level client

Server Operations
–
–
BACKUP STGPOOL – chunks decompressed first
BACKUPSET and EXPORT
•
•
Data is decompressed during a backup set generation or an export
operation, if the node for which the backup set or export being generated is
TSM 6.1 or prior.
For TSM 6.2 and later clients, the chunks in backup set volumes and export
volumes are not decompressed.
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication Reporting
 B/A Client
Total number of objects inspected:
2
Total number of objects backed up:
2
Total number of objects deduplicated: 2
Total number of bytes inspected:
Total number of bytes transferred:
7.22 MB
0 B
Deduplication reduction:
100%
Total data reduction ratio:
100%
 API Client
– Extended “dsmEndSendObjExOut”
– DP Products not exploiting deduplication reporting at GA release
 Server
– Additional ANEnnnn client messages written to activity log
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication Reporting
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication Integrity
 Hash collisions
–
–
–
–
Generate MD5 hash on the entire file
Save on server
Return to client on restore / retrieve
The file is not restored if the 2 digests do not match
 Protection against security attacks
– Such as:
• Spoof hash values to gain access to chunks
• Store hash values that do not match data
– Protect by:
•
•
•
•
SHOW commands show limited output
Chunk size stored along with hash value
Query must be followed by store, or protocol violation
SET DEDUPLICATIONVERIFICATIONLEVEL nn (nn>0)
– If failure to verify, NODE updated to DEDUPLICATION=SERVERONLY
– Message ANR2695E issued
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Considerations
 Separate session for chunk lookups on the server
– One extra session per TSM Client consumer or per TSM API thread
– Consider increasing server MAXSESSIONS parameter on the server
 File space, backup, or archive deletions may result in the deduplication
cache being out of synch with the server
– TSM Client detects the condition and resets deduplication cache
– Failed transaction is re-tried
 The same deduplication cache cannot be used from multiple processes
– Second process will not use the deduplication cache and will query the server for chunks
– Different processes can use separate cache files by using DEDUPCACHEPATH option
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Client-Side Data Deduplication - Considerations
 ENABLEDEDUPCACHE option defaults:
– YES for BA client – BA client can recovery from invalid cache
– NO for API – minimize txn aborts due to hash lookup failure
 DP products not updated for client-side dedup
– DP for Oracle and DB2 may start multiple sessions
– DP for SQL and Exchange legacy backups can be large transactions
– DP for SQL and Exchange VSS backups use BA client
 Initial storage pool must be sequential disk
– Should not migrate objects to tape quickly
•
Migration can invalidate client dedup cache
– Consider device class mountlimit setting
– Number of volumes must support number of backup sessions
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Incremental Windows System State Backup
 Current Problem:
– Any change in the system state cause a full backup of the system state
– While Windows 2003 system was relatively inactive, Windows 2008 system state
changes much more often
– Windows Vista/7/2008 can generate 50,000+ files and over 7 GB of system state data
 What is New:
– TSM only backs up changed files in the “system files” component of system state
(typically a very large component)
– TSM will use backup grouping functions to manage the versions of the system files
– No new externals except that customer can assign System State backup to a
Management Class with “mode=absolute” to force a full backup
– Can use TSM 6.2 client with TSM 5.5 or TSM 6.1 server
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Incremental Windows System State Backup
 Backup Versioning:
Backup Day 1
Backup Day 2
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
V6.2 Summary
 Performance
–
–
–
–
–
Limits on TXNBYTELIMIT and MOVESIZETHRESH
TSM for ERP: Improve TSM Handling of Very Large Objects
Simultaneous Write During Storage Pool Migration
Concurrent Access To Centera Volumes
Progressive Incremental for Windows System State
 Function
–
–
–
–
–
Windows B/A Client Support for GPFS 3.3
Automatic Deployment for Windows Clients
Security: In Flight Data Encryption Using SSL
Client-side Data Deduplication
Windows Passthru Driver
 Virtualization
– Microsoft Hyper-V Full Guest Virtual Machine Backup Using Volume Shadow Copy
Services (VSS)
– Auto Discovery Of New Vmware Guests
– VCB Gen II - VMware vStorage for Data Protection API integration
© 2009 IBM Corporation
Chicago Tivoli Storage Manager Users Group Meeting April 2010
Thank You
146
Getting to the New DB2 Database
© 2009 IBM Corporation
© 2010 IBM Corporation