Transcript CMG 2007 - Merrill Consultants

UK CMG 2008
H. W. "Barry" Merrill, PhD
Merrill Consultants
Dallas, TEXAS
www.mxg.com
Tuesday May 20, 2008
Remote from
Emerald Island, NC
A. MXG Support for SAS Version 9.2
1. Compatible, No ERRORs, New Warnings
B. MXG and WPS
1. Current Status of MXG and WPS
2. Run Time Comparison
3. Revisions to SAS Clone Newsletter Article
4. Summary
5. QA Steps successfully compiled and run
C. CPU Parked Time Metric
D. zIIP and zAAP Metrics
E. Daily CPU cost of Performance Measurement
A. Support for SAS Version 9.2: COMPATIBLE, no ERRORS, new WARNings
All MXG Versions execute WITHOUT error with SAS Version V9.2.
V9.2 libraries can be read/written by SAS V8.2 or V9.1.3, & vice versa.
SAS V9.2 Phase I Foundation Level z/OS + ASCII was tested.
These old MXG Versions WILL print a new SAS V9.2 WARNING,
that sets CC=4 (condition/return code), but that warning is harmless
(to MXG code), and all MXG output SAS datasets are correct,
even with that warning and return code.
MXG Version 26.02 eliminates this SAS V9.2 WARNING internally,
by enabling new OPTION VARLENCHK=NOWARN to suppress
the creation of both the warning and the condition code.
So the ONLY exposure with older MXG Versions under V9.2
Is only on z/OS, and ONLY if condition code tests are used in
your MXG job streams.
This new-in-SAS V9.2 "MULTIPLE LENGTHS OF A VARIABLE“
Warning message surfaced in MXG delivered code in two cases:
-The shortening of the LENGTH of a numeric variable, but only if
when the LENGTH statement precedes SET/MERGE/UPDATE.
This occurs in VMXGSUM where the fixed-length-8 variables
output by PROC MEANS were reduced to 4-bytes, prior to option
KEEPLEN. The VMXGSUM utility is used in all MXG summarization,
like ASUMxxxx and TRNDxxxx, many ANALxxxx members, and
in summarizing RMFINTRV and CICINTRV programs in BUILDPDB.
It is pervasive in MXG. This case was eliminated in MXG 26.02 by
relocating its LENGTH statement to after the SET.
- The JOIN of multiple datasets (SET MON.JOBS TUE.JOBS ...)
where a variable has different lengths in different datasets, and
the first dataset’s variable has shorter length.
This also occurs in VMXGSUM, when multiple input datasets are
combined, like TRENDing, where TREND had short LENGTHs,
but the "NEWTREND" internally has fixed, longer LENGTHS.
MXG 26.02 adds KEEPLEN option to PROC MEANS to eliminate.
MXG Version 26.02 code eliminates the SAS V9.2 WARNING internally,
but also enables OPTION VARLENCHK=NOWARN to suppress BOTH
the warning and the condition code, for all your SAS programs, as long
as you use the MXGSASV9/CONFIGV9 MXG initialization.
Without VARLENCHK=NOWARN, EVEN with MXG 26.02, WARNING
can OCCUR:
- If you have tailoring members in "USERID.SOURCLIB" from old
MXG versions, that need the same code revisions.
-
In any user-written SAS programs, but this could actually be a
valid warning that a variable was unintentionally truncated.
or, at any time in the future, the WARNING can still occur:
-
When an MXG Version that changed a variable LENGTH is
installed, subsequent WEEKLY or MONTHLY jobs create the
WARNING because some PDB's have the old length and some
have the new length, when those multiple datasets are joined.
Previous to V9.2, length could be changed with no WARNING.
Between MXG 24.24 and 25.25 1206 variables were changed.
Hot Fix VARLENCHK for Phase I will be included in Phase II.
Changes 26.060 and 26.065 have all the V9.2 details.
B.MXG and WPS
1.
Current status of MXG testing under WPS.
a. What has been tested successfully:
MXG QA compile-only (dummy INFILEs) TYPExxxx and TYPSxxxx members
This exercised the MXG DATA-step Code for compile errors,
and created DOCVER to compare the contents of WPS-built
datasets (variables, LENGTHs, LABELs, FORMATs).
Steps 1 thru 36 of the QAJOBXX were tested.
Default BUILDPDB with 480MB SMF File.
Tailored BUILDPDB with IMACEXCL with 480 MB SMF File.
TYPETPMX with small file - verified 72 position FORMATs worked.
TYPENTSM with test file - verified open-system-style MXG code.
b. WPS is often updated daily, and as errors were encountered in
MXG tests on z/OS and Windows/XP, a new release was generated
which did correct each error that could be fixed short-term.
The current release is World Programming System 2.3 of May 4.
c. MXG Version 25.11 is required for testing under WPS; its updates
eliminate the need for user modifications to MXG Software, and new
WPS-specific members (CONFIGW2, MXGWPSV2, JCLINSTW, AUTOEXEW)
were created.
d. Facilities used by MXG Software not yet in WPS for z/OS:
INFILE CCHHR option - needed only for TYPEEREP (EREP,SYS1.LOGREQ)
VIEWS
- for I/O performance, used in VMXGSUM, which
is invoked over 60 time in QABUILDPDB, and
also in all ASUMxxxx and TRNDxxxx members.
Eliminates a full pass of the input data.
PROC CONTENTS
- minor, does not report High Block Used size.
e. Facilities used by MXG Software not yet in WPS for z/OS and Windows:
INFILE with ftp access method - performance slower, more disk space
SAS/GRAPH, SAS/STAT, SAS/ETS, etc.
Currently WPS supports the Base DATA Step Support and a few PROCs,
including PRINT, MEANS, CONTENTS GPLOT, GCHART, with more planned.
f. Output differences - minor
PROC MEANS printed output - some values printed with one digit more
resolution by WPS.
- no error in output values spot checked
- prevents automated output comparisons
g. Untested - most due to complexity or time to set up multiple inputs:
DATA Libraries on TAPE
WEEKBLDT, MNTHBLD special TAPE format to DISM, Mod, etc.
TESTTRND, TESTANAL
- requires extensive test data
ANALxxxx
- requires test data
ASUMxxxx
- except ASUMJOBS, ASUM70PR were tested.
UTILxxxx
- specialized utilities for MXG Tech Support
Internal SORT
- on z/OS, internal SAS SORT may be required
when BY list variables don't fit in first
4096 bytes. Have not confirmed in WPS.
NFS Files
- not tested.
Broken VBS files
- not tested.
h.
Migration issues
- see WPS Migration Instructions:
On z/OS, copy all archived SAS Data Libraries on DASD (including HSM
migrated) to tape (Sequential) format with the SAS System first.
WPS can read SAS Data Libraries in SEQUENTIAL format on tape or DASD.
WPS cannot read DASD format SAS Data Libraries
i. Customer reports:
Several MXG customers have been running their BUILDPDB on z/OS.
Early tests had to make source-level-changes to a few MXG members.
Most had relatively simple BUILDPDBs with minimal tailoring.
j. MXG Support Position for execution under WPS product:
MXG 25.11+ and WPS 2.2 or later are required.
Your current MXG Software License Agreement states:
Merrill provide continuous product support for MXG Software:
When error conditions are the results of errors in MXG code,
they will be corrected; error is defined: SAS® execution
of MXG code produces either a return code or an ABEND.
If you encounter an execution error testing MXG under WPS:
You should report the error to WPS Technical Support for
initial investigation.
If WPS support believes the error is an MXG problem, they can
contact MXG, or they can refer you to contact [email protected].
If the error can be replicated under the SAS System, it will
be corrected per our license terms.
For data errors, (INPUT STATEMENT EXCEEDED RECORD etc)
Contact MXG Support by sending the FULL job log, and maybe
- to ftp the raw data file that caused the error
- or to ftp your USERID.SOURCLIB (MXG tailoring) libraries
2.A.
Run time comparisons of SAS and WPS - NOVEMBER 2007:
BUILDPDB,ASUMJOBS,ASUM70PR with 448 MB SMF File
z/OS Comparison
CPU TCB
minutes
Elapsed
minutes
----------- Compressed ---------------Work Size
PDB Size
CICSTRAN Size
SAS:
3.88
5.20
209 MB
55 MB
59 MB
WPS:
10.67
18.50
233 MB
59 MB
67 MB
Windows/XP Comparison
SAS:
2.18
Note 1
65 MB
63 MB
WPS:
2.71
Note 1
78 MB
70 MB
Note 1:
Neither WPS nor SAS provide a way to track maximum work
space on ASCII, except for free space monitoring with DIR
2.B. Comparison of SAS V9.2 and WPS 2.3 on z/OS. May 2008:
BUILDPDB and ASUMs with 448 MegaByte input SMF file;
z/OS Step Totals:
SAS V9.2
WPS 2.3
RATIO
Total CPU time
mm:ss
03:45
sec
225
mm:ss
08:30
sec
510
2.26
Total Elapsed time
08:00
480
16:58
1018
2.10
Total EXT Memory
Total SYS Memory
Total Memory
104
11
115
MegaBytes
MegaBytes
MegaBytes
188 MegaBytes
504 MegaBytes
692 MegaBytes
SMF Data Step
Elapsed time
SMF Read Rate
mm:ss
01:12
sec
72
6.22 MB per sec
mm:ss
03:27
sec
207
2.16 MB per sec
2.85
3.
Revisions to SAS Clones article in MXG Technical Newsletter FIFTY:
WPS is no longer vapour-ware.
The company had bent over backwards to provide corrections.
When originally written in JAVA several years ago, WPS performed
so poorly, with run times at least ten times worse than the
current release, that JAVA is no longer used in the product.
So the offload of your SAS CPU time to a zAAP won’t happen.
Items listed in sub-paragraphs i, ii., iii., and iv. have all been
addressed to my satisfaction, with the exception of the items that
are listed above in this note.
Pricing has been significantly reduced from those original IBM
prices. As an example, in late 2007, an MXG site in the USA was
quoted an IBM price for a 21 Value Unit system (about 1000 MIPS)
of $42,000 first year and $8,400 for renewals.
WPS can be licensed through IBM or through World Programming;
their home page is at
http://www.teamwpc.co.uk
4.
Summary:
Most of MXG has compiled successfully under WPS on both z/OS and
Windows/XP.
BUILDPDB has compiled and processed SMF data on both z/OS and
Windows/XP.
Not all of MXG has been tested with data.
WPS is still in development; January’s 2.2 had five errors.
This week’s new WPS 2.3 claims to have corrected them.
WPS on z/OS requires thrice the CPU and Elapsed Run Time of SAS.
It is claimed that WPS 2.3 has improved z/OS performance.
WPS on Windows/XP run times are similar to SAS run times.
Disk Space required on both platforms are similar.
So, it is safe now to test MXG under WPS.
5.
QA Steps successfully compiled and executed with dummy input.
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
CREATE FORMAT LIBRARY
RUN TESSNT
RUN TESSIBM
RUN TESSIBM1
RUN TESSIBM2
RUN TESSIBM3
RUN TESSUSER
RUN TESSUSR1
RUN TESSOTHR
RUN TYPSCMHM
RUN BUILDPD3+ASUMS
RUN BUILDPDB+ASUMS
RUN TYPERMFV
RUN TYPECMFV+TYPEMVCI
RUN TESSHSM
RUN TESSFACO
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
STEP
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
RUN TESSVM
RUN TESSCRAY
RUN TESSHPCS
RUN TESSUNIA
RUN TESSUNIK
RUN TESSQAPM
RUN TESSQACS
RUN TESSTUX
RUN TESSPW
RUN ASUMPRTR
RUN TYPEIMS7
RUN TESSIMSD
RUN COPYSTEP
RUN TESSTRND - partially completed, not WPS fault.
RUN TESSMNTH
RERUN TYPE-S FOR NON PROC SORT FOR DATASET LABEL
RUN CROSSREF
RUN DOCVER
- output DOCVER file is identical.
C.
CPU Parked Time Metric.
PR/SM data for LCPUADDR 5 in dataset TYPE70PR:
"Online Duration"
|======================SMF70ONT===========================|
299.97
Online, "Parked"
Online, "Dispatched or Not Parked"
|=====CPUPATTM======|======= (SMF70ONT-CPUPATTM) =========|
103.22
196.75
Online
Online
"Dispatched"
"Not Parked"
|====LCPUPDTM====|======PATWAITM======|
96.80
99.96
MVS data for CPUID 5 in dataset TYPE70:
SMF70WAT
|=ORIGWAIT=|
0.0000
This data for LCPUADDR=5 shows a CP engine that was parked for 103
seconds of that 5 minute interval. RMF subtracts the SMF70PAT
parked duration from the SMF70ONT online duration to calculate the
Percent MVS Busy value. In this interval, ORIGWAIT was zero for
this engine, as MVS never entered the wait state on that engine,
so RMF calculates the MVS busy percent as:
PCTMVSBY= 100*(SMF70ONT-ORIGWAIT-SMF70PAT)/(SMF70ONT-SMF70PAT);
PCTMVSBY= 100*( 299
-0
-103)
/(299
-103) = 100%
The IBM calculation of the PCTCPUBY, the LPAR CPU busy percent, is
NOT altered by parked time; PCTCPUBY=32%, calculated as
PCTCPUBY= 100*(LCPUPDTM/SMF70ONT); = 100 * (96 / 299 );
The "PATWAITM", the time when the CP engine is "not parked", is the
time when this CP engine could/should have been parked, but was
still online and not-dispatched, because the algorithm to park a
CPU only executes occasionally. It is not created in TYPE70PR.
D.
zIIP and zAAP measurements when they are faster than CPU engines.
When specialty engines are faster than the speed of your CPs, there
is a normalization factor to convert the recorded seconds to their
NORMALIZED (EQUIVALENT) time, as if they had executed on your CPs.
In all MXG workload datasets, TYPE72GO and RMFINTRV, (and TYPE30),
all time variables for zIIPs and zAAPS are NORMALIZED seconds, and
all of the service units are segregated by engine type.
However, the IBM RMF reports present these data quite differently.
This system has the normalization factor, R723NFFS=569/256=2.222,
that is, one second of zIIP is equal to 2.222 seconds of CP time.
================================================================
MXG Dataset TYPE72GO dataset values:
================================================================
SERVICE
3,932,091
CPUUNITS
1,793,920
ZIPUNITS
2,137,167
CPUTCBTM
178.92
CPUZIPTM
213.16
RMF WORKLOAD REPORT:
Under "SERVICE TIMES", the RMF "CPU" value of 392.9 seconds is the
total of the real CPU time on CP engines, plus the NORMALIZED CPU
time on the zIIP and zAAP engines; it is NOT the CPU "TCB" time.
( 392.9 = 178.92 + 213.16
"RMF CPU" = CPUTCBTM + CPUZIPTM )
But also under "SERVICE TIMES", the RMF "IIP" (zIIP) value of 96.1
seconds is the UN-NORMALIZED, raw, seconds on the zIIP engine.
And the RMF "AAP" value for zAAPs is also the UN-NORMALIZED value.
And under "SERVICE", the RMF "CPU" value of 3931K is the TOTAL
SERVICE units from CPs, zIIPs, and zAAPs.
REPORT BY: POLICY=OWL
WORKLOAD=CSSDDF
TRANSACTIONS
---SERVICE---SERVICE TIMES
AVG
0.23
IOC
0
CPU
392.9
MPL
0.23
CPU
3931K
SRB
0.0
ENDED
51
MSO
0
RCT
0.0
END/S
0.01
SRB
0
IIT
0.0
#SWAPS
0
TOT
3931K
HST
0.0
EXCTD
0
/SEC
1092
AAP
N/A
AVG ENC 0.23
IIP
96.1
---APPL
CP
AAPCP
IIPCP
%--4.98
0.00
0.07
AAP
IIP
N/A
2.67
While the workload datasets have normalized CPU time, in all of the
"hardware" datasets, TYPE70, TYPE70PR, ASUM70PR etc., the CPU times
for the zIIP and zAAP engines are the raw seconds of CPU Dispatch
Time on those engines, and is NOT normalized. As a result, then,
the total ZIPACTTM recorded in TYPE70 for the above system for the
day was 10,887 seconds, but the total CPUZIPTM in TYPE72GO for the
day was 23,079 seconds.
Those 10,887 raw hardware seconds would be 24,190 normalized seconds
so the zIIP capture ratio at this site is 23079/24190 = 95.4%.
E. The CPU cost of performance monitoring and capacity planning.
One MXG user reports they currently write 500 GB of SMF data per day
or an average rate of 6 MegaBytes per second across all platforms.
They dump SMF multiple times each day, and build multiple "PDB's"
throughout the day, and run many ad hoc analysis reports as well.
They have SMF, RMF, OMEGAMON, and NETVIEW monitors consuming CPU.
The daily total CPUTM for each of their workloads were:
OMEGAMON
MXG JOBS
RMF III
RMF I
SMF DUMPS
MONITORS
SMF ASID
28:56:37
19:05:01
12:20:05
6:29:11
4:12:30
2:17:10
0:29:16
TOTAL CPUTM
73:30:50
= 2% of 3744 HOURS with 156 CPs
Thus this sites total daily cost of 74 CPU hours is an average
use of 3 CP engines all day long, but with 156 CP Engines, that
is ONLY 2% of the installed CP engine capacity, for the entire
CPU cost of
performance monitoring,
data collection,
building PDBs,
archiving,
and all MXG daily reporting and ad hoc analysis.
to read 518GB of SMF data
838K TMC records
902K DASD DSN records (DCOLLECT)
27K DASD VOL records (DCOLLECT)
The following graph shows the hourly CPU consumption of these
workloads.