Transcript DB2 Memory management - GSE Young Professionals

DB2 Memory management
DB2 GSE 01/12/2011
Agenda
 Overview
 Statistics
 EDMPOOL
 DBDPOOL
 Dynamic Statement Cache (DSC)
 Bufferpools
TS / Mainframe Systems
|
16/07/2015
|
2
Overview
PLXA
SYA1
MQP1
IMP1
DBP1
SYA3
MQP3
IMP3
DBP3
DBPI
CF
CF
SYA5
PLXB
PLXC
PLXT
SYA7
MQP7
IMP7
DBP7
SYB2
MQC2
IMC2
DBC2
SYB7
MQC7
IMC7
DBC7
SYB2
SYC2
DBD2
MQC2
DBT2
IMC2
DBQ2
DBC2
SYT1
DBCF
MQP2
IMP2
DBP2
SYA2
MQP4
IMP4
DBP4
SYA4
SYA6
MQC3
IMC3
DBC3
MQP8
IMP8
DBP8
SYA8
MQC1
IMC1
DBC1
SYB1
MQC8
IMC8
DBC8
SYB8
C
F
MQN1
IMN1
DBN1
DBI1
DBCF
DBZ1
C
F
MQQ1
IMQ1
DBQ1
MQT1
MQC3
IMT1
IMC3
DBT1
DBC3
MQD1
MQC1
IMD1
IMC1
DBD1
DBC1
SYC1
MQQ7
IMQ7
DBQ7
MQT7
IMT7
DBT7
MQD7
IMD7
DBD7
SYC7
MQN2
IMN2
DBN2
DBN3
SYT2
C
F
TS / Mainframe Systems
|
16/07/2015
|
3
Statistics
 SMF 100 Statistics trace records
 Format and Load in DB2 tabel with BMC Mainview
 Detail information (10 minute interval)
 Summary by hour
 System statistics
 Buffer statistics
 3 months history
 Table field descriptions in User Guide
 Additional descriptions in SDSNIVPD member DSNWMSGS
TS / Mainframe Systems
|
16/07/2015
|
4
EDMPOOL
 EDM Pools – V9
TS / Mainframe Systems
|
16/07/2015
|
5
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
Pages 4Kb
EDMPOOL
EDM Pool Below Pages
40000
35000
30000
25000
20000
FREEPAGES
USEDPAGES
15000
10000
5000
0
DATE HOUR
TS / Mainframe Systems
|
16/07/2015
|
6
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
Pages 4Kb
EDMPOOL
EDM Pool Above Pages
3500
3000
2500
2000
PKGPAGEAB
CTPAGESAB
1500
1000
500
0
DATE HOUR
TS / Mainframe Systems
|
16/07/2015
|
7
2011-10-31_16
2011-10-30_17
2011-10-29_18
2011-10-28_19
2011-10-27_20
2011-10-26_21
2011-10-25_22
2011-10-24_23
2011-10-24_00
2011-10-23_01
2011-10-22_02
2011-10-21_03
2011-10-20_04
2011-10-19_05
2011-10-18_06
2011-10-17_07
2011-10-16_08
2011-10-15_09
2011-10-14_10
2011-10-13_11
2011-10-12_12
2011-10-11_13
2011-10-10_14
2011-10-09_15
2011-10-08_16
2011-10-07_17
2011-10-06_18
2011-10-05_19
2011-10-04_20
2011-10-03_21
2011-10-02_22
2011-10-01_23
2011-10-01_00
Pages 4Kb
EDMPOOL
SKEL Pool Pages
40000
35000
30000
25000
20000
SKELPOOLFREE
SKELPOOLPAGES
15000
10000
5000
0
DATE HOUR
TS / Mainframe Systems
|
16/07/2015
|
8
TS / Mainframe Systems
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
EDMPOOL
Percentage CT LOAD
100%
98%
96%
LOADCTS
CTREQS
94%
92%
90%
|
16/07/2015
|
9
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
Number of loads by Minute
EDMPOOL
LOAD of CT by minute
30
25
20
15
LOADCT/MIN
10
5
0
TS / Mainframe Systems
|
16/07/2015
|
10
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
EDMPOOL
Percentage PT LOAD
100%
98%
96%
PKGLOAD
PKGREQS
94%
92%
90%
TS / Mainframe Systems
|
16/07/2015
|
11
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
Number of loads by Minute
EDMPOOL
LOAD of PT by minute
350
300
250
200
LOADPKG/MIN
150
100
50
0
TS / Mainframe Systems
|
16/07/2015
|
12
-2000
2011-10-31_00
2011-10-30_00
2011-10-29_00
2011-10-28_00
2011-10-27_00
2011-10-26_00
2011-10-25_00
2011-10-24_00
2011-10-23_00
2011-10-22_00
2011-10-21_00
2011-10-20_00
2011-10-19_00
2011-10-18_00
2011-10-17_00
2011-10-16_00
2011-10-15_00
2011-10-14_00
2011-10-13_00
2011-10-12_00
2011-10-11_00
2011-10-10_00
2011-10-09_00
2011-10-08_00
2011-10-07_00
2011-10-06_00
2011-10-05_00
2011-10-04_00
2011-10-03_00
2011-10-02_00
2011-10-01_00
Pages 4Kb
Dynamic Statement Cache
Dynamic Statement Cache Pages
14000
12000
10000
8000
6000
STMTPOOLFREE
STMTPOOLPAGES
4000
2000
0
DATE HOUR
TS / Mainframe Systems
|
16/07/2015
|
13
TS / Mainframe Systems
|
2011-10-31_20
2011-10-31_00
2011-10-30_04
2011-10-29_08
2011-10-28_12
2011-10-27_16
2011-10-26_20
2011-10-26_00
2011-10-25_04
2011-10-24_08
2011-10-23_12
2011-10-22_16
2011-10-21_20
2011-10-21_00
2011-10-20_04
2011-10-19_08
2011-10-18_12
2011-10-17_16
2011-10-16_20
2011-10-16_00
2011-10-15_04
2011-10-14_08
2011-10-13_12
2011-10-12_16
2011-10-11_20
2011-10-11_00
2011-10-10_04
2011-10-09_08
2011-10-08_12
2011-10-07_16
2011-10-06_20
2011-10-06_00
2011-10-05_04
2011-10-04_08
2011-10-03_12
2011-10-02_16
2011-10-01_20
2011-10-01_00
Percentage
Dynamic Statement Cache
Dynamic Statement Cache Hit Ratio
100
90
80
70
60
50
HIT
40
30
20
10
0
16/07/2015
|
14
2011-10-31_16
2011-10-30_17
2011-10-29_18
2011-10-28_19
2011-10-27_20
2011-10-26_21
2011-10-25_22
2011-10-24_23
2011-10-24_00
2011-10-23_01
2011-10-22_02
2011-10-21_03
2011-10-20_04
2011-10-19_05
2011-10-18_06
2011-10-17_07
2011-10-16_08
2011-10-15_09
2011-10-14_10
2011-10-13_11
2011-10-12_12
2011-10-11_13
2011-10-10_14
2011-10-09_15
2011-10-08_16
2011-10-07_17
2011-10-06_18
2011-10-05_19
2011-10-04_20
2011-10-03_21
2011-10-02_22
2011-10-01_23
2011-10-01_00
Pages 4Kb
DBDPOOL
DBD Pool Pages
8000
7000
6000
5000
4000
DBDPOOLFREE
DBDPAGES
3000
2000
1000
0
DATE HOUR
TS / Mainframe Systems
|
16/07/2015
|
15
2011-10-31_06
2011-10-30_08
2011-10-29_10
2011-10-28_12
2011-10-27_14
2011-10-26_16
2011-10-25_18
2011-10-24_20
2011-10-23_22
2011-10-23_00
2011-10-22_02
2011-10-21_04
2011-10-20_06
2011-10-19_08
2011-10-18_10
2011-10-17_12
2011-10-16_14
2011-10-15_16
2011-10-14_18
2011-10-13_20
2011-10-12_22
2011-10-12_00
2011-10-11_02
2011-10-10_04
2011-10-09_06
2011-10-08_08
2011-10-07_10
2011-10-06_12
2011-10-05_14
2011-10-04_16
2011-10-03_18
2011-10-02_20
2011-10-01_22
2011-10-01_00
Number of loads by Minute
DBDPOOL
LOAD of DBD by minute
16
14
12
10
8
LOADDBD/MIN
6
4
2
0
TS / Mainframe Systems
|
16/07/2015
|
16
EDMPOOL - DBDPOOL - DSC
 Triggers for critical problems:
 EDMFAILS (QISEFAIL) : EDM pool full failures
 SKELPOOLFULL (QISEKFAL) : SKELETON POOL FULL failures
 STMTPOOLFULL (QISECFAL) : below the 2GB bar EDM pool
statement cache full failures
 STMTPOOLABFULL (QISESFAL) : above the 2GB bar EDM pool
statement cache full failures
 DBDPOOLFULL (QISEDFAL) : DBD pool full failures
TS / Mainframe Systems
|
16/07/2015
|
17
Bufferpools
 Main separation
 The first bufferpool by page size is for Catalog, Directory and DB2
Tools
 The last bufferpool by page size is for Sort and Work
 By Type
 AG
 BNP
 INFOCENTER
 Application Tablespaces are in bufferpools with an even number
 Application indexes are in bufferpools with an odd number
 By Size
 By access
TS / Mainframe Systems
|
16/07/2015
|
18
Bufferpools
 Initial setup
TS / Mainframe Systems
|
16/07/2015
|
19
Bufferpools

BP setup for specific reason
TS / Mainframe Systems
|
16/07/2015
|
20
Bufferpools
 Data separation BP setup
TS / Mainframe Systems
|
16/07/2015
|
21
Bufferpools
 Problems
 How to decide on random – Sequential
 Growth of object
 Changes in behavior
 Difference in load
 Example on BP3 – BP5
 BP3 – 20000 pages – Index Random
 BP5 – 20000 pages – Index Sequential > 30%
TS / Mainframe Systems
|
16/07/2015
|
22
Bufferpools
 Bufferpool Getpages
BP3 - BP5
3500000
3000000
Getpage
2500000
2000000
BP3 GETP
BP5 GETP
1500000
1000000
500000
0
8
9
10
11
12
13
14
15
16
17
Hour
TS / Mainframe Systems
|
16/07/2015
|
23
Bufferpools
 Bufferpool SYNC and ASYNC I/O
BP3 - BP5 I/O
3000000
3500
BP3 –Index Random
3000
2500000
BP5 –Index Sequential
2500
2000000
BP3 ASYNC
BP3 SYNC
BP5 ASYNC
BP5 SYNC
BP5
BP3
2000
1500000
1500
1000000
1000
500000
500
0
0
8
9
10
11
12
13
14
15
16
17
Hour
TS / Mainframe Systems
|
16/07/2015
|
24
Bufferpools
 Desired Setup
 Only possible to change for Bufferpools based on data separation
 In memory objects
 Small objects
 High getpage rate
 No I/0 after first read
 Hot spot objects
 Objects with a high re-read factor (i.e. > 20 getpages/read)
 Remaining objects
 Specific Bufferpools
TS / Mainframe Systems
|
16/07/2015
|
25
Bufferpools
 How to move a object to another Bufferpool
 DB2 table containing
 list of objects
 New Bufferpool
 Current Bufferpool
 Rexx procedure
 Check objects that need to be changed
 Check if they are in use
 Issue STOP AT COMMIT
 ALTER object
 START OBJECT
 CHECK for STOPP
 Done during weekly maintenance window
TS / Mainframe Systems
|
16/07/2015
|
26
Bufferpools
 When moving objects verify impact
 Original Bufferpool
 New Bufferpool
 Statistics for one index for one day:
DATE
GETPAGE
GETPAGE SEC
1/11/2011
864.350
10
2/11/2011
12.184.492
141
3/11/2011
13.559.082
174
4/11/2011
12.465.366
144
7/11/2011
13.772.394
159
8/11/2011
13.979.690
162
9/11/2011
12.333.078
143
10/11/2011
16.769.820
194
11/11/2011
2.769.708
32
14/11/2011
14.053.658
165
15/11/2011
13.745.612
159
16/11/2011
12.798.930
148
17/11/2011
14.403.748
167
18/11/2011
13.310.282
154
21/11/2011
14.021.386
163
22/11/2011
13.949.742
161
23/11/2011
12.444.360
144
24/11/2011
13.769.470
159
SYNC
SYNC_SEC
158.884
2
4.523.002
52
4.978.822
58
4.563.904
53
5.161.940
60
5.071.538
59
4.510.772
52
5.053.646
58
61.356
1
5.150.240
60
4.994.538
58
4.651.662
54
5.223.732
60
4.738.454
55
5.134.192
59
5.025.202
58
4.476.582
52
4.999.572
58
TS / Mainframe Systems
|
16/07/2015
|
27
Bufferpools
 PAGEFIX
 Fix - Release is done when memory page is changed
 High I/O intensity
 Pages read = SYNC + ASYNC + GBP read
 Pages written = SYNC + DEFERRED + GBP Write
 Is activated or deactivated at bufferpool allocation
 Up to 8% reduction
 Disadvantage
 Page can never be reused by any other process
 Pages might be necessary for DUMP Processing
TS / Mainframe Systems
|
16/07/2015
|
28
Bufferpools
 Actions already done
 Move some objects to best fitting Bufferpool
 In memory Bufferpool BP7/BP8
 PGSTEAL FIFO
 Follow-up of I/O
 PAGEFIX BP16 with 50000 pages
 Merge two sets of Bufferpool
 BP11 + BP13 each 50000 pages to one of 100000
 BP12 + BP14 each 50000 pages to one of 100000
TS / Mainframe Systems
|
16/07/2015
|
29
Bufferpools
 Results of actions already done
 Move objects to best fitting Bufferpool
 Higher Getpage rate, 2500 to over 4000 a second at peak hours
for in memory bufferpool BP7 with same size
 In memory Bufferpool BP7/BP8
 FIFO had no measurable CPU difference in applications due to
divers workload
 PAGEFIX BP16 with 50000 pages
 no measurable CPU difference in applications due to divers
workload
 Merge two sets of Bufferpool
 No impact seen in application Follow-up
 small improvements/degradation for some Transactions
 Improvements for batch with large workingset
TS / Mainframe Systems
|
16/07/2015
|
30
Bufferpools
 System page residency time
 buffer pool size / total pages read per second
 Random page residency time
 Max
 System page residency time
 Bufferpool size * (1-VPSEQT/100) / sync pages read per second
 More info can be found in the Redpaper DB2 9 for z/OS: Buffer Pool
Monitoring and Tuning of Mike Bracey
http://www.redbooks.ibm.com/redpieces/abstracts/redp4604.html?Open&pdfbookmark
TS / Mainframe Systems
|
16/07/2015
|
31
11
20 - 09
11 -2
6
20 - 09 _0
11 -2 8
6
20 - 09 _1
11 -2 3
7
20 - 09 _0
11 -2 8
7
20 - 09 _1
11 2 3
8
20 - 09 _0
11 -2 8
8
20 - 09 _1
11 2 3
9
20 - 09 _0
11 -2 8
9
20 - 09 _1
11 -3 3
0
20 - 09 _0
11 -3 8
0
20 - 10 _1
11 -0 3
3
20 - 10 _0
11 0 8
3
20 - 10 _1
11 -0 3
4
20 - 10 _0
11 0 8
4
20 - 10 _1
11 -0 3
5
20 - 10 _0
11 -0 8
5
20 - 10 _1
11 -0 3
6
20 - 10 _0
11 -0 8
6
20 - 10 _1
11 0 3
7
20 - 10 _0
11 -0 8
7
20 - 10 _1
11 -1 3
0
20 - 10 _0
11 -1 8
0
20 - 10 _1
11 -1 3
1
20 - 10 _0
11 1 8
1
20 - 10 _1
11 -1 3
2
20 - 10 _0
11 1 8
2
20 - 10 _1
11 -1 3
3
20 - 10 _0
11 -1 8
3
20 - 10 _1
11 -1 3
- 1 4_
0- 08
14
_1
3
20
Seconds
Bufferpools
System Residency Time
3.500
3.000
2.500
2.000
Merge BP
TS / Mainframe Systems
BP11
BP12
BP13
1.500
BP14
1.000
500
0
|
16/07/2015
|
32
11
20 - 09
11 -2
6
20 - 09 _0
11 -2 8
6
20 - 09 _1
11 2 3
7
20 - 09 _0
11 -2 8
7
20 - 09 _1
11 -2 3
8
20 - 09 _0
11 -2 8
8
20 - 09 _1
11 -2 3
9
20 - 09 _0
11 2 8
9
20 - 09 _1
11 -3 3
0
20 - 09 _0
11 -3 8
0
20 - 10 _1
11 -0 3
3
20 - 10 _0
11 -0 8
3
20 - 10 _1
11 0 3
4
20 - 10 _0
11 -0 8
4
20 - 10 _1
11 -0 3
5
20 - 10 _0
11 -0 8
5
20 - 10 _1
11 -0 3
6
20 - 10 _0
11 0 8
6
20 - 10 _1
11 -0 3
7
20 - 10 _0
11 0 8
7
20 - 10 _1
11 -1 3
0
20 - 10 _0
11 -1 8
0
20 - 10 _1
11 1 3
1
20 - 10 _0
11 -1 8
1
20 - 10 _1
11 1 3
2
20 - 10 _0
11 -1 8
2
20 - 10 _1
11 -1 3
3
20 - 10 _0
11 -1 8
3
20 - 10 _1
11 -1 3
- 1 4_
0- 08
14
_1
3
20
Seconds
Bufferpools
Random Residency Time
4.000
3.500
3.000
2.500
Merge BP
2.000
TS / Mainframe Systems
BP11
BP12
BP13
BP14
1.500
1.000
500
0
|
16/07/2015
|
33
Bufferpools
 PAGEFIX BP16
 High I/O during peak period is more then 200 pages/sec
 Not measurable due to workload
DATE
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
24.11.2011
HOUR BPSYNCRD
BPSIO
Total
0
365.438 2.909.625 3.275.063
1
259.643 2.220.904 2.480.547
2
387.075 4.817.165 5.204.240
3
238.023 2.920.779 3.158.802
4 12.930.036 2.977.275 15.907.311
5
75.754 1.879.117 1.954.871
6
217.389 4.925.515 5.142.904
7
202.868 2.873.951 3.076.819
8
370.740 2.585.147 2.955.887
9
704.297 3.420.917 4.125.214
10
1.012.645 2.197.940 3.210.585
11
1.002.819 4.446.559 5.449.378
12
1.056.198 2.403.122 3.459.320
13
836.299 2.082.528 2.918.827
14
674.737 2.059.295 2.734.032
15
817.531 3.139.465 3.956.996
16
693.589 2.065.780 2.759.369
17
1.032.719 3.921.567 4.954.286
18
645.822 2.490.670 3.136.492
19
800.455 6.031.868 6.832.323
20
363.395 2.422.383 2.785.778
21
557.267 3.661.068 4.218.335
22
236.879 2.339.994 2.576.873
23
192.829 2.394.598 2.587.427
25.674.447 73.187.232 98.861.679
TS / Mainframe Systems
|
16/07/2015
|
34
BPREADS
BPFPAGESRD
BPPAGEINRD
10.10.2011 05
59527
2118
0
0
10.10.2011 06
1705543
73959
40362
49987
24.11.2011
23.11.2011
21.11.2011
20.11.2011
18.11.2011
17.11.2011
15.11.2011
14.11.2011
12.11.2011
11.11.2011
09.11.2011
07.11.2011
06.11.2011
04.11.2011
03.11.2011
01.11.2011
31.10.2011
29.10.2011
28.10.2011
BPGETPAGE
HOUR
26.10.2011
25.10.2011
23.10.2011
DATE
21.10.2011
20.10.2011
18.10.2011
50000
17.10.2011
60000
15.10.2011
14.10.2011
12.10.2011
11.10.2011
09.10.2011
08.10.2011
06.10.2011
05.10.2011
Pages
Bufferpools
BP16 PAGEIN by Hour
40000
30000
BPPAGEINRD
20000
10000
0
TS / Mainframe Systems
|
16/07/2015
|
35
Questions
TS / Mainframe Systems
|
16/07/2015
|
36