Transcript Slide 1

Introduction to the LOGPARSE Macro
GASUG Atlanta, GA
July 18th, 2007
Our Services
 Integration Consulting
 Marketing Automation and Optimization
 Marketing Campaign Management, Fulfillment, and
Response Analysis
 Risk & Analytics
 Analytical Modeling
 Data Warehousing
 Business Intelligence Solutions
 CRM
 Database Solutions
 Web Development Services
Our Locations: 5 Main Hubs + International
3. New York
5. UK & Europe
2. Charlotte
4. Dallas
1. Atlanta
International
Corporate Location:
4080 McGinnis Ferry Rd
Bldg 100, Suite 101
Alpharetta, GA 30005
We have completed projects
in Korea, Singapore,
Malaysia, United Kingdom,
Netherlands, Sweden,
Germany, and Peru.
Our Atlanta Lab
Always Seeking Talent
Send inquiries to:
[email protected]
Today’s Topic
 Introduce the LOGPARSE macro to capture
FULLSTIMER statistics
 Demonstrate a technique for profiling very large SAS
applications
 Provide code for creating a Performance Dashboard
…Capture Tuning Statistics
…Not Tune SAS Applications
LOGPARSE MACRO
 Provided by SAS, experimential with SAS 9.1
 Captures FULLSTIMER statistics from the SAS Log
and stores them in a SAS Dataset
• For every SAS Step
 Available on the SAS Scalability & Performance
Community Site
www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html
www.support.sas.com/rnd/scalability/tools/fullstim/logparse.zip
What’s Inside logparse.zip
 Four files in logparse.zip
 %LOGPARSE()
• Extracts the statistics from the SAS LOG(s)
 %PASSINFO()
• Optional, provides session information to the %LOGPARSE()
macro.
 %MVSNAME()
• Called by %PASSINFO() for MVS only
 Readme.txt
• A good place to start
Getting Started with %LOGPARSE
 OPTIONS FULLSTIMER; as the first line for the
program you wish to collect statistics
 %PASSINFO; optionally on the second line
 Run the program to generate the log(s)
 Create another program to use %LOGPARSE and
collect the statistics
%include logparse;
%logparse( yourProgram.log, yourPerfData, OTH );
proc print data=yourPerfData;
run;
Big Picture
LOG
FILE(S)
YOUR
LOGPARSE
PROGRAM
Statistics
Dataset
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
%LOGPARSE Syntax
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
 saslog is the name of the SAS LOG to parse, requires the .log extension
 outds is the output SAS dataset for the parsed LOG results
 system is the 3-CHAR code for the system on which %LOGPARSE is run.
not specified, the default value is the system %LOGPARSE is executing on.
• MVS - z/OS, OS/390, or MVS
• ALP - OpenVMS Alpha
• VMS - VMS
• OTH - All other OSs, like Windows
 pdsloc identifies the PDS member where the SAS LOG is stored on MVS
system. See the readme.txt file.
 append=no specifies whether to append the results to an existing file or
overwrite
• NO – creates a new SAS file based on the outds parameter
• YES – appends the output to file specified by outds or creates a new file if
the file does not exist
If
The STATISTICS Dataset
 The statistics captured by %LOGPARSE are
dependent on the NOTES: produced by FULLSTIMER
and vary by system.
 Important non-statistics columns to understand
• logfile – name of the log file
• stepname – DATA, SORT, FORMAT, PRINTO, SAS,
initialization.
NOTE: DATA statement used (Total process time):
NOTE: The SAS System used:
• stepcnt – sequence number applied to the order of the
step within the log.
 Windows Statistics Columns
•
•
•
realtime
usertime
systemtime
•
•
•
obsin
obsout
varsout
•
memused
Big Picture
LOG
FILE(S)
YOUR
LOGPARSE
PROGRAM
Statistics
Dataset
%LOGPARSE( saslog, outds, system, pdsloc, append=no )
Example
options fullstimer;
%let defaultPath=C:\Documents and Settings\Administrator\My Documents\My SAS Code;
%include "&defaultPath\timerUtils\passinfo.sas";
%passinfo;
%include "&defaultPath\census\ipumsFormats.inc";
%include "&defaultPath\census\convertToSAS.inc";
%include "&defaultPath\census\splitData.inc";
%include "&defaultPath\census\subsetMetro.inc";
%include "&defaultPath\census\hhRollup.inc";
libname ipumsdat "&defaultPath\census";
/*----------------------------------------------------------------------------+
Main Program
+----------------------------------------------------------------------------*/
%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);
%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);
%splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);
%subsetMetro("Atlanta", 2003 2004 2005 2006);
%hhRollup(2003 2004 2005 2006)
The Bad News
 A dataset of statistics is produced – map that back to
your program/log…. OR
stepcnt stepname realtime usertime systime cputime obsin obsout varsout memused
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
initialization
DATA
OPTIONS
OPTIONS
OPTIONS
FORMAT
FORMAT
DATA
DATA
DATA
SORT
DATA
SORT
DATA
SORT
DATA
SORT
DATA
DATA
DATA
DATA
SAS
00:00.5 .
00:00.0
00:00.0
00:00.0
00:00.0
00:00.7
00:00.0
01:28.6
00:55.0
00:01.6
00:00.0
00:01.4
00:00.0
00:01.5
00:00.0
00:01.4
00:00.0
00:00.2
00:00.0
00:00.0
00:00.0
02:31.3
.
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:32.1
00:15.8
00:01.3
00:00.0
00:01.3
00:00.0
00:01.3
00:00.0
00:01.2
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:53.0
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:03.5
00:03.7
00:00.2
00:00.0
00:00.2
00:00.0
00:00.2
00:00.0
00:00.1
00:00.0
00:00.0
00:00.0
00:00.0
00:00.0
00:08.3
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
00:00.2 .
.
.
.
.
.
.
.
.
.
.
.
.
3052
848875
848875
1564
1564
1651
1651
1922
1922
2427
2427
600
619
742
921
3052
848875
1564
1564
1651
1651
1922
1922
2427
2427
1564
1651
1922
2427
.
.
.
.
.
.
.
.
.
21
220
880
9
9
9
9
9
9
9
9
10
10
10
10
.
161
13
13
13
182
88
333
506
203
282
203
282
203
316
203
367
209
209
209
209
3382
The Good News
Use PRINTTO to Write Separate Logs
filename fmtLog "&defaultPath\census\ipumsFormtas.timingDemo.log";
filename cvrtLog "&defaultPath\census\convertToSAS.timingDemo.log";
filename splitLog "&defaultPath\census\splitData.timingDemo.log";
filename substLog "&defaultPath\census\subsetMetro.timingDemo.log";
filename hhrupLog "&defaultPath\census\hhRollup.timingDemo.log";
/*----------------------------------------------------------------------------+
Main Program
+----------------------------------------------------------------------------*/
proc printto log=fmtLog new; run;
%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);
MODULE.PROGRAM.LOG
proc format cntlin=ipumsdat.fmmiller_ermas_com_001;
run;
proc printto log=cvrtLog new; run;
%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);
proc printto log=splitLog new; run;
%splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);
proc printto log=substLog new; run;
%subsetMetro("Atlanta", 2003 2004 2005 2006);
proc printto log=hhrupLog new; run;
%hhRollup(2003 2004 2005 2006);
run; proc printto; run;
Use PROC PRINTTO to Write
Separate Log Files
ERMAS Performance Toolkit
 Use our toolkit package to create a visual performance
dashboard (WINDOWS ONLY)
 Name each log in the format module.program.log for
each subset of the program to be analyzed
 Use the perfDash.sas program to analyze the logs
/*----------------------------------------------------------------------------+
User Parameters
rootDir - location of the SAS utilities, see installation instructions below
logDir - directory where the SAS LOGs are located
program - name of the SAS program for which to generate the dashboard
+----------------------------------------------------------------------------*/
%let rootDir = %str(C:\GASUG2007);
%let logDir = %str(C:\GASUG2007\census);
%let program = timingDemo; /* case sensitive */
Resources
 How to get the Ermas Performance Toolkit
http://www.ermas.com/downloads.htm
• GASUG2007.ZIP file contains four directories:
 timerUtils – location of the perfDash.sas program
 census – log files to analyze
 myLibrary – utility library
 utils – more utilities.
 SAS Scalability & Performance Community
www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html
 Great SUGI Paper on the LOGPARSE Macro
http://www2.sas.com/proceedings/sugi30/219-30.pdf